Data Engineer

Iris's direct client, a leader Payroll, is looking to hire a strong Databricks Platform Engineer for a Long Term Contractual role. Job title: Data & AI Platform Engineer — (Databricks + AWS) Location: Hybrid (3 days onsite, Roseland, NJ) Duration: 18 Months Skills: Databricks, Pyspark, Python, Genie Space Or Genie Agent Job Description About the Role Own the Databricks lakehouse layer and AI/LLM platform for Finance Insights. You'll design the data architecture on Databricks Unity Catalog, build Genie Spaces for natural language analytics, and integrate LLMs to power intelligent financial workflows — all running on AWS. Responsibilities • Design and build Delta Lake tables on Databricks (bronze/silver/gold medallion architecture) for financial datasets (AR, GL, invoices, collections) • Build and manage Databricks Genie Spaces — configure semantic models, write GENIE-compatible SQL, and tune NL-to-SQL behavior for finance use cases • Develop PySpark and Python-based data pipelines (Databricks Jobs + Workflows) for ERP data ingestion, transformation, and aggregation • Integrate Databricks Model Serving with LLMs (Claude, GPT-4, or custom fine-tuned models) for AI-driven insights, anomaly detection, and financial forecasting • Build RAG pipelines using Databricks Vector Search over financial documents, contracts, and GL notes • Expose Databricks SQL Warehouse endpoints to Python API services and the React frontend • Manage Unity Catalog governance — row-level security, column masking, data lineage for finance compliance • Deploy infrastructure on AWS (S3 as Databricks external storage, IAM roles, VPC networking, Secrets Manager) Required Skills • Databricks (PySpark, Delta Lake, Jobs/Workflows, SQL Warehouses) — 3+ years • Databricks Unity Catalog and governance • Databricks Genie Spaces (semantic layer, NL analytics) • Python 3.11+ for data engineering and API integration • AWS (S3, IAM, VPC, Secrets Manager, optionally Lambda/Glue) • LLM integration: Databricks Model Serving, MLflow, or direct API calls to Claude/OpenAI • SQL (advanced — window functions, CTEs, query optimization)