Senior Data Engineer

Senior Data Engineer - 6 month contract - London/Hybrid - £750 I’m working with a leading consultancy that is looking to appoint a Technical Lead to take ownership of the architecture and delivery of a large-scale Databricks Lakehouse platform. This is a hands-on technical leadership role, working within a complex production environment to expand an established data platform, build new integrations and pipelines, and support the delivery of new reporting capabilities. The successful candidate will combine strong Databricks and Spark expertise with the ability to make architectural decisions, lead engineers and communicate effectively with both technical and business stakeholders. Key Responsibilities • Lead the Databricks platform strategy and architecture across the Lakehouse, from ingestion through to serving. • Design and build scalable Lakehouse and streaming data pipelines using PySpark and medallion architecture. • Own technical and architectural decisions across an established production data platform. • Build new data integrations and pipelines to support additional data sources, products and reporting requirements. • Work closely with upstream technical teams to manage dependencies and align design decisions. • Translate business requirements into clear technical solutions and delivery plans. • Define and promote engineering standards, governance and best practices across the platform. • Mentor engineers and provide technical leadership across the team. • Ensure the platform supports performant and scalable downstream BI and analytics, particularly Power BI. Essential Skills & Experience • Strong hands-on Databricks and Spark/PySpark experience. • Proven experience designing and delivering enterprise-scale data or Lakehouse architectures. • Strong experience working within Azure data environments. • Experience with technologies such as ADLS Gen2, Event Hubs and Key Vault. • Infrastructure as Code experience using Terraform and/or Bicep. • Strong understanding of medallion architecture, DataOps, streaming and batch processing. • Experience implementing CI/CD practices for data pipelines. • Experience building and productionising data platforms or data products in complex enterprise environments. • Strong understanding of how Lakehouse architectures support Power BI, semantic models and scalable reporting. • Comfortable working within regulated, high-pressure production environments. • Excellent stakeholder management and communication skills, with the ability to translate complex technical information for different audiences. • Previous experience providing technical leadership and mentoring engineers.