SC Cleared Databricks Data Engineer – Azure Cloud

Job Title: SC Cleared Databricks Data Engineer – Azure Cloud Contract Type: 12 month contract Day Rate: Up to £400 a day inside IR35 Location: Remote or hybrid (as agreed) Start Date: January 5th 2026 Clearance required: Must be holding active SC Clearance We are seeking an experienced Databricks Data Engineer to design, build, and optimise large-scale data workflows within the Databricks Data Intelligence Platform. The role focuses on delivering high-performing batch and streaming pipelines using PySpark, Delta Lake, and Azure services, with additional emphasis on governance, lineage tracking, and workflow orchestration. Client information remains confidential. Key Responsibilities • Build and orchestrate Databricks data pipelines using Notebooks, Jobs, and Workflows • Optimise Spark and Delta Lake workloads through cluster tuning, adaptive execution, scaling, and caching • Conduct performance benchmarking and cost optimisation across workloads • Implement data quality, lineage, and governance practices aligned with Unity Catalog • Develop PySpark-based ETL and transformation logic using modular, reusable coding standards • Create and manage Delta Lake tables with ACID compliance, schema evolution, and time travel • Integrate Databricks assets with Azure Data Lake Storage, Key Vault, and Azure Functions • Collaborate with cloud architects, data analysts, and engineering teams on end-to-end workflow design • Support automated deployment of Databricks artefacts via CI/CD pipelines • Maintain clear technical documentation covering architecture, performance, and governance configuration Required Skills and Experience • Strong experience with the Databricks Data Intelligence Platform • Hands-on experience with Databricks Jobs and Workflows • Deep PySpark expertise, including schema management and optimisation • Strong understanding of Delta Lake architecture and incremental design principles • Proven Spark performance engineering and cluster tuning capabilities • Unity Catalog experience (data lineage, access policies, metadata governance) • Azure experience across ADLS Gen2, Key Vault, and serverless components • Familiarity with CI/CD deployment for Databricks • Solid troubleshooting skills in distributed environments Preferred Qualifications • Experience working across multiple Databricks workspaces and governed catalogs • Knowledge of Synapse, Power BI, or related Azure analytics services • Understanding of cost optimisation for data compute workloads • Strong communication and cross-functional collaboration skills