Senior Data Engineer

Senior Data Engineer / Data Platform Engineer (Databricks) Contract | Outside IR35 Remote (UK/EU) £550–£650 per day (DOE) We’re partnering with a forward-thinking Data & AI consultancy that is helping to scale a Databricks Lakehouse platform. They are looking for a Senior Data Engineer / Data Platform Engineer who can operate across both Data Engineering & Platform Engineering. This is an end-to-end hands-on role. What You’ll Be Doing: • Designing and building end-to-end data pipelines using Databricks (PySpark / SQL / Delta Lake) • Developing and managing robust, scalable Lakehouse architectures • Implementing Delta Live Tables (DLT) for reliable, testable, production-grade pipelines • Optimising performance using: • Working with Unity Catalog to implement Centralised governance • Leveraging Databricks Workflows / Jobs for orchestration • Building CI/CD pipelines for data & platform deployments (Terraform, GitHub Actions, Azure DevOps etc.) • Enabling real-time and streaming pipelines (Structured Streaming, Auto Loader) • Contributing to MLOps workflows, integrating models into production pipelines • Collaborating with Data Scientists, Analysts, and Platform teams to improve developer experience and platform usability Key Skills & Experience: • Strong experience building pipelines with PySpark / Spark SQL • Deep understanding of Delta Lake • Databricks Platform Expertise • Hands-on experience with: Unity Catalog (governance & security), Spark Declarative Pipelines, Databricks Workflows / Jobs • Strong understanding of Lakehouse architecture principles • Performance & Optimisation • Platform / DevOps Engineering, Infrastructure as Code, CI/CD pipelines • Strong experience in at least one - Azure, AWS Very Nice to Have: • Databricks certifications (Data Engineer Associate / Professional) • Experience with Feature Store / MLflow • Exposure to Databricks AI / Mosaic AI / Model Serving If you’re interested in building high-performance Databricks platforms at scale, send your CV to cv@primus-connect.com or click apply.