Lead Data Engineer

Title: Lead Data Engineer Location: Warwick, UK Duration: 06 Months Possible extension Work Type: Hybrid Pay: £350 to £460 per day Umbrella (inside IR35) Background • Client is is maturing and embedding its existing Data & Insights platform, which is on the Azure stack. We have a combination of strategic Databricks and legacy Synapse engineering. • We have an existing programme team which requires a skilled individual to lead the engineering resource. This person will need to embed our Databricks architecture, ensure correct configuration and documentation, whilst also overseeing the ingestion of new data sources. • It is a live platform, and the role may also need to provide technical guidance to support engineers to ensure any production issues are understood and resolved efficiently. • This is a UK based role. The team predominantly work across the UK (with some off-shore team members). Occasional onsite meeting attendance. Responsibilities • Design and delivery of modern Azure and Databricks Data platform aligning to agreed technical designs and best practice. • Responsible for data engineering across all layers of our data platform. There is currently a mix of legacy synapse and strategic Databricks. • Responsible for leading work on related DevOps - branch supervision, merging • In depth Databricks DBA • Responsible for promotion of data pipelines and associated notebooks through the various environments (development, fof, production) • Responsible for the correct configuration and documentation of Databricks • Enhancing existing monitoring & control framework • Provide technical leadership & coaching for a small group of Data Engineers Required skills: • Extensive “hands on” Databricks & Synapse experience • Extensive Azure Dev Ops experience • Ideally SQL and PySpark in a Databricks environment • Experience of using Databricks in a federated cloud environment • Leadership of small engineering teams • Configuration management and documentation • Certified Data Bricks Professional Engineer