Databricks Developer – Medicare

Databricks Developer – Medicare • Advantage (Payer) • Work Model: Remote (U.S.-based candidates only) • Level: Senior • Domain: Healthcare Payer / Medicare Advantage Overview • Our client, a leading Medicare Advantage health plan, is seeking an experienced Databricks Developer to build and • operate data solutions on the Databricks lakehouse platform, supporting claims, eligibility, and member/provider • data used across Medicare Advantage reporting and compliance. This is a Databricks platform-focused role: the • ideal candidate lives inside Databricks day to day — notebooks, Delta Live Tables, Workflows, Unity Catalog, and • cluster/job tuning — rather than a general data engineer who happens to use Databricks as one of several tools. • Candidates should be comfortable working independently in a remote setup and available to start promptly. Key Responsibilities • Build, deploy, and maintain data pipelines natively within Databricks, using notebooks, Delta Live Tables • (DLT), and Databricks Workflows/Jobs for orchestration. • Develop and optimize Spark (PySpark/Scala/SQL) logic for processing large-scale Medicare Advantage • claims (EDI 837/835), eligibility, and provider data within the Databricks environment. • Manage and evolve the Delta Lake medallion architecture (bronze/silver/gold) to ensure clean, reliable, • well-governed data. • Configure and maintain Unity Catalog for data governance, lineage, and access control across PHI/HIPAA- • regulated datasets. • Tune cluster configurations, job performance, and cost efficiency (autoscaling, cluster policies, Photon, • job/task-level optimization) within Databricks. • Use Databricks SQL and SQL warehouses to support ad hoc analysis, validation, and reconciliation of • Medicare Advantage claims and eligibility data. • Monitor pipeline health, troubleshoot job failures, and resolve performance issues directly within the • Databricks platform (Jobs UI, cluster logs, DLT event logs). • Manage code and deployments via Databricks Repos and CI/CD integration (Databricks Asset Bundles, Azure DevOps/Git). • Apply PHI/HIPAA-compliant data handling and role-based access practices throughout all Databricks workspace and pipeline configurations. • Document Databricks pipeline design, workspace structure, and job configurations. Required Experience & Skills • 5+ years of hands-on Databricks experience — this is a Databricks-first role; deep platform fluency is the primary requirement. • Strong command of Spark (PySpark/Scala) and Databricks SQL for pipeline development. • Hands-on experience building and managing Delta Live Tables (DLT) pipelines. • Practical experience with Unity Catalog for data governance, access control, and lineage. • Experience with Databricks Workflows/Jobs for orchestration and scheduling. • Demonstrated ability to tune cluster and job performance (autoscaling, cluster policies, Photon, cost optimization) within Databricks. • Experience with Databricks Repos and Git-based CI/CD for notebook/pipeline deployment. • 3–10+ years of healthcare payer domain experience, ideally with Medicare Advantage data (claims ingestion, eligibility logic, EDI 837/835, CMS-aligned reporting). • Working knowledge of PHI/HIPAA-regulated data handling and role-based security within a data platform. • Comfortable working within an Azure or equivalent cloud environment (ADLS, workspace/networking basics) as the underlying infrastructure for Databricks. • Strong problem-solving skills and ability to work independently in a remote environment. • U.S.-based, remote-capable (U.S. resident, citizen, or Green Card holder). Nice to Have • Databricks Certified Data Engineer (Associate or Professional). • Experience with MLflow or Databricks' ML/AI tooling. • Exposure to Azure Data Factory, Power BI, or Snowflake in a supporting/downstream capacity. • Experience migrating legacy ETL workloads into Databricks. • Prior experience in a client-facing or consulting environment. • Experience with EHR-adjacent systems (e.g., Epic Clarity) or provider-side data in addition to payer data. Ideal Candidate Profile "Senior Databricks Developer who lives in the platform day to day — Delta Live Tables, Unity Catalog, Workflows, and Spark performance tuning — with real Medicare Advantage claims and eligibility experience, and a track record of running governed, production-grade lakehouse pipelines end to end within Databricks.