Data Engineer (Databricks) - SC Clearance Required

Senior Data Engineer – Databricks & Big Data Pipelines (SC Cleared is required for this role) Rate: £450-550 per day inside IR35 We're looking for a Senior Data Engineer – Databricks & Big Data Pipelines to join our team, working in our UK remote working division. This position requires active UK Security Check (SC) clearance and offers the opportunity to build advanced, large-scale data engineering solutions for an innovative analytics platform. You will design, develop and optimize distributed data pipelines on Databricks and cloud ecosystems while ensuring performance, scalability, and compliance. The role combines technical depth with client-facing delivery as part of an agile and collaborative environment. Responsibilities • Design, develop and optimize end-to-end data pipelines using Databricks, Spark/PySpark and Python • Lead the full pipeline lifecycle: analysis, architecture, development, testing, deployment and monitoring • Implement ingestion, normalization and standardization workflows for high-volume and diverse data sources • Ensure data platform architecture supports performance, security, and compliance requirements • Collaborate cross-functionally with engineering, product teams and client stakeholders during project delivery • Support ML readiness for ingestion and transformation pipelines where required • Promote data platform best practices across security, governance and cloud adoption strategies Mandatory Requirements • Active SC Clearance • Expertise in Databricks, including Delta Lake and Unity Catalog • Strong background in Python, PySpark and advanced SQL • Proven experience in AWS cloud platforms for data engineering • Track record of building high-performance, large-scale data solutions in production environments • Ability to lead pipeline lifecycle (design, deploy, monitor) autonomously • Strong communication skills for stakeholder collaboration and client presentations SkillSet Overview Must Have: • Databricks (Delta Lake, Unity Catalog, workflows) • Python Data Engineering Technologies (including PySpark) • SQL • AWS Nice to Have: • AI and ML Ops • PostgreSQL • OpenSearch • Microsoft Azure • Claude Code (AI productivity tooling) • Docker / Kubernetes • REST APIs • Cloud-native ML pipeline deployment • Familiarity with managing data platform releases, monitoring and cost governance • Domain understanding in supply chain analytics, compliance or public sector data Experience integrating AI tools into engineering workflows