DataBricks Engineer

DataBricks Lead Engineer Raritan office. Local NJ consultants preferred. Hybrid Long term contract Role Summary We are seeking a Databricks Engineer to build of a new Databricks solution that ingests source files and constructs a semantic/serving layer that powers a Power BI dashboard. A semantic layer already exists; the new solution must conform to the same structure so that a replicated Power BI solution works seamlessly against it with no rework to reports, measures, or relationships. The engineer owns the Databricks data and semantic layer and partners with a dedicated Power BI developer/team, who own the dashboard build, to deliver the end solution. Key Responsibilities • Lead the design and delivery of file-ingestion pipelines in Azure Databricks (batch and/or streaming) that land and process incoming files into Delta Lake using a Medallion architecture (Bronze → Silver → Gold). • Design the Gold/serving layer to replicate the existing semantic model exactly — matching table names, grain, column names, data types, keys, and relationships so the existing Power BI dataset can be repointed without redesign. • Expose the model through Databricks SQL endpoints (Direct Query) so the Power BI developer can connect and validate the dashboard with no redesign. • Collaborate with the Power BI developer/team as the data-side owner — align on the semantic contract (star schema, measures, RLS expectations), support their dashboard build, and resolve any data or model issues to land the end deliverable. • Document the semantic mapping and work with the Power BI team to reconcile the replicated dashboard's output against the existing solution to prove parity. Required Skills • AWS Databricks, Delta Lake, Medallion architecture, Unity Catalog • Python and strong SQL • Databricks SQL endpoints and a working understanding of Power BI semantic modeling (star schema, DAX, incremental refresh, RLS) sufficient to define and hand off the semantic contract • Databricks Certified Data Engineer (Professional preferred) Regards, Jagannath Gaddam jagannath.gaddam@quantumIntegrators.com