Lead Data Engineer

Bristol, Darlington, London, Manchester, Wolverhampton Job Summary Here at the Ministry of Housing, Communities and Local Government (MHCLG), we work on things that make a real difference to people’s lives. Whether it's through the homes we live in, the work of our local councils, or the communities we’re all a part of, our work is at the top of the political agenda. We have ambitious and far-reaching outcomes to achieve this year and, if you’re thinking of joining us, there’s never been a more exciting time. We have circa 3,500 staff who are based in 20 offices across the UK. The Digital Planning Programme is delivering a once-in-a-generation transformation of England’s planning system. The Common Data Platform is a federated data platform enabling teams to get to work with data faster. Working with large datasets form a variety of sources, we are creating modern, digital, and data-driven tools and services to support local authorities, developers, communities, and central government. As a Lead Data Engineer, you’ll play a key role in building and managing the data infrastructure that underpins this transformation. You’ll lead the design of reusable data pipelines, open data services, and making planning and land data more transparent, accessible, and interoperable. This role combines hands-on engineering with technical leadership, supporting cross-government efforts to make planning data more open, standardised, and integrated across the planning lifecycle. You can explore our platform here Planning Data, and as we work in the open, you can find out what we do here at Digital Land. Job Description As a Lead Data Engineer, you'll: • Lead the design, development, and maintenance of modern data pipelines and platforms that ingest, process, and publish planning data from local authorities and external providers • Collaborate with product managers, policy specialists, developers, and analysts to turn complex planning data into usable, accessible services for multiple user groups • Model and transform data in line with open standards, working toward consistent, high-quality data structures • Build and maintain cloud-native data infrastructure using platforms such as AWS or Azure (e.g., Glue, Lambda, S3, Redshift, or equivalent) • Ensure robust metadata management, version control, and data lineage to support transparency and auditability • Coach junior data engineers and help build internal capability in data engineering and best practice • Ensure compliance with data governance, protection, and security standards, and work in line with FAIR data principles and open data expectations • Contribute to the department’s open-source and open-data commitments by developing public-facing datasets and APIs • Manage and maintain key pipelines supporting our platform to ensure service reliability As a Lead Data Engineer, You'll • Lead the design, development, and maintenance of modern data pipelines and platforms that ingest, process, and publish planning data from local authorities and external providers • Collaborate with product managers, policy specialists, developers, and analysts to turn complex planning data into usable, accessible services for multiple user groups • Model and transform data in line with open standards, working toward consistent, high-quality data structures • Build and maintain cloud-native data infrastructure using platforms such as AWS or Azure (e.g., Glue, Lambda, S3, Redshift, or equivalent) • Ensure robust metadata management, version control, and data lineage to support transparency and auditability • Coach junior data engineers and help build internal capability in data engineering and best practice • Ensure compliance with data governance, protection, and security standards, and work in line with FAIR data principles and open data expectations • Contribute to the department’s open-source and open-data commitments by developing public-facing datasets and APIs • Manage and maintain key pipelines supporting our platform to ensure service reliability Person specification As a Lead Data Engineer, You'll Have • Proven experience designing and implementing end-to-end data pipelines using modern programming languages and tools such as DuckDB, Parquet, Python, SQL, Airflow, PostGIS, Spark, Git • Strong experience with cloud-based data architecture and infrastructure (e.g. AWS Glue, S3, Lambda, Redshift, or Azure equivalents) • Experience working with large, structured, semi-structured, and geospatial datasets, and understanding data quality and standardisation challenges • Ability to lead technical discussions, influence data design decisions, and communicate effectively with non-technical stakeholders • Experience working in agile, cross-disciplinary delivery teams • Strong understanding of data ethics, privacy, and security frameworks, including experience with anonymisation and responsible data use Alongside your salary of £61,374, Ministry of Housing, Communities and Local Government contributes £17,780 towards you being a member of the Civil Service Defined Benefit Pension scheme. Find out what benefits a Civil Service Pension provides. At MHCLG we offer many benefits that range from tailored career pathways and flexible working to MyLifestyle Childcare Voucher and Cycle to Work Schemes. For more information, please click here. Artificial intelligence Artificial intelligence can be a useful tool to support your application, however, all examples and statements provided must be truthful, factually accurate and taken directly from your own experience. Where plagiarism has been identified (presenting the ideas and experiences of others, or generated by artificial intelligence, as your own) applications may be withdrawn and internal candidates may be subject to disciplinary action. Please see our candidate guidance (opens in a new window) for more information on appropriate and inappropriate use. Selection process details Artificial Intelligence Artificial Intelligence can be a useful tool to support your application, however, all examples and statements provided must be truthful, factually accurate and taken directly from your own experience. Where plagiarism has been identified (presenting the ideas and experiences of others, or generated by artificial intelligence, as your own) applications may be withdrawn and internal candidates may be subject to disciplinary action. Please see our candidate guidance for more information on appropriate and inappropriate use. Application and Selection The application process will be split into 2 stages, testing the following Success Profiles: Behaviours , Experience , Technical Please ensure your CV does not contain any personally identifiable information. Note: We do not consider direct CV applications. All applications must be submitted via the provided application link. Stage 1: Sift (CV) Experience: Your CV will be assessed using the experience success profile. Most of our campaigns utilise multiple assessors and so it is possible that each of your answers would be viewed by different assessors. In the event that we receive a large number of applications, we may conduct an initial sift using the lead sift question listed in the advert. Candidates who pass the initial sift may be progressed to a full sift, or progressed straight to interview. Lead sift question (Technical): Tell us about a time when you made large volumes of varied data more open, accessible, or interoperable. How did you approach designing and implementing technical solutions, applying data standards, and engaging with stakeholders to ensure the data was usable and valuable beyond your immediate team? Stage 2: Interview • Behaviours: "Leadership", "Delivering at a pace", "Working together", "Changing and Improving" • Experience: Experience questions will be based around the essential skills and criteria as listed in the job description. • Technical: Technical questions will be based around the essential skills and criteria as listed in the job description. Sift and Interview Dates Sifting is envisaged to take place the week commencing 5th January 2026. Interviews are envisaged to take place the week commencing 26th January 2026 and are currently being held remotely via videocall. This could be subject to change. Group 1 Digital & Data Roles MHCLG has implemented the Digital and Data capability framework for Group 1 roles. Applicants that are successful and have been offered a position will be required to complete a capability assessment after the interview. MHCLG will honour completed capability assessments for this role from other Government Departments for existing civil servants on level transfer only. Please provide a copy of your capability assessment to the Hiring Manager when applying. If you have any queries on pay, please contact the Hiring Manager. Each experience or technical skill is assessed between 1-3, representing working towards, at or above the job level requirements. You are awarded a proficiency level accordingly, and you will be given opportunity to annually re assess your capability and potentially increase your overall remuneration, through payment of allowances, depending on level of assessed capability. Candidates moving from another government department have the option to retain their current basic pay if this is within our pay band for the relevant grade. Grade 7 Group 1 Digital and Data salary • The basic pay for this role will be between £66,929 - £81,112 (London), £61,374 - £75,581 (National). Candidates will usually be recruited to the median of the payscale. • A digital allowance may also be payable depending on the level of assessed capability, in order to meet the MHCLG overall targeted remuneration rate for the specific grade, location and capability rating. • For applicants in receipt of existing allowances, we will assess each case individually, but we would seek not to exceed the MHCLG overall targeted remuneration rate for the specific grade, location and capability rating for existing civil servants, the usual policy on level transfer and promotion will apply, subject to the statements above. How We Recruit Find out everything you need to know before applying here.. • Applying • Sifting • Interview • Interview Results & Feedback • Reserve List • Near Miss • Civil Service Grades • We are a DCS, RIS & GPTWV employer • Reasonable Adjustments How To Apply What you need to know before applying. • Security Clearance Requirements • Civil Service Nationality Requirements • Right to Work • Artificial Intelligence • Civil Service Code and Recruitment Principles • CV Declaration • Sponsorships • Salary and Grade • Existing Civil Servants • Conflict of Interest • Location and Flexible Working • Fixed Term Contracts • Internal Fraud Database - Internal Fraud Register • Appeals and Complaints • Conflict of Interest Security Clearance Requirements National Office: BPSS + SC London Office: BPSS + SC Feedback will only be provided if you attend an interview or assessment. Security Successful candidates must undergo a criminal record check. Successful candidates must meet the security requirements before they can be appointed. The level of security needed is security check (opens in a new window).See our vetting charter (opens in a new window). People working with government assets must complete baseline personnel security standard (opens in new window) checks. Successful candidates must undergo a criminal record check. Successful candidates must meet the security requirements before they can be appointed. The level of security needed is security check (opens in a new window).See our vetting charter (opens in a new window). People working with government assets must complete baseline personnel security standard (opens in new window) checks. Nationality requirements This Job Is Broadly Open To The Following Groups • UK nationals • nationals of the Republic of Ireland • nationals of Commonwealth countries who have the right to work in the UK • nationals of the EU, Switzerland, Norway, Iceland or Liechtenstein and family members of those nationalities with settled or pre-settled status under the European Union Settlement Scheme (EUSS) (opens in a new window) • nationals of the EU, Switzerland, Norway, Iceland or Liechtenstein and family members of those nationalities who have made a valid application for settled or pre-settled status under the European Union Settlement Scheme (EUSS) • individuals with limited leave to remain or indefinite leave to remain who were eligible to apply for EUSS on or before 31 December 2020 • Turkish nationals, and certain family members of Turkish nationals, who have accrued the right to work in the Civil Service Further information on nationality requirements (opens in a new window) Working for the Civil Service The Civil Service Code (opens in a new window) sets out the standards of behaviour expected of civil servants. We recruit by merit on the basis of fair and open competition, as outlined in the Civil Service Commission's recruitment principles (opens in a new window). The Civil Service embraces diversity and promotes equal opportunities. As such, we run a Disability Confident Scheme (DCS) for candidates with disabilities who meet the minimum selection criteria. The Civil Service also offers a Redeployment Interview Scheme to civil servants who are at risk of redundancy, and who meet the minimum requirements for the advertised vacancy. The Civil Service Code (opens in a new window) sets out the standards of behaviour expected of civil servants. We recruit by merit on the basis of fair and open competition, as outlined in the Civil Service Commission's recruitment principles (opens in a new window). The Civil Service embraces diversity and promotes equal opportunities. As such, we run a Disability Confident Scheme (DCS) for candidates with disabilities who meet the minimum selection criteria. The Civil Service also offers a Redeployment Interview Scheme to civil servants who are at risk of redundancy, and who meet the minimum requirements for the advertised vacancy. Diversity and Inclusion The Civil Service is committed to attract, retain and invest in talent wherever it is found. To learn more please see the Civil Service People Plan (opens in a new window) and the Civil Service Diversity and Inclusion Strategy (opens in a new window). This vacancy is part of the Great Place to Work for Veterans (opens in a new window) initiative. Once this job has closed, the job advert will no longer be available. You may want to save a copy for your records. Contact point for applicants Job Contact • Name : Digital Data Jobs • Email : Digitaldatajobs@communities.gov.uk Recruitment team • Email : Digitaldatajobs@communities.gov.uk Further information About If you are unsure about any part of the process or require additional information about the post to enable you to progress your application, you should speak to the Resourcing team - Recruitment@communities.gov.uk. Candidates can appeal at any stage of the recruitment process if they believe there has been: • a procedural irregularity • an infringement of the Civil Service equal opportunities policy • exceptional circumstances which were not notified to the interview panel which might have affected performance on the day. It is important to note that these are appeals about the process not the decision. In the first instance, an appeal should be directed to the MHCLG Resourcing Hub at Recruitment@communities.gov.uk. If the MHCLG Resourcing Hub is unable to satisfactorily resolve your complaint, you may contact put your complaint in writing to the Civil Service Commission, Room G8,1 Horse Guards Road, London, SW1A 2HQ or by email to: info@csc.gov.uk