Principal Data Engineer

Principal Data Engineer - Secure Digital Platform (SDP) Data & Integration Location: Reading (Hybrid Working Available) Security Clearance: Active SC Clearance Required Industry Experience: Aerospace, Defence, MOD or Defence Digital environments essential About the Role We are seeking a highly capable Principal Data Engineer to join the Secure Digital Platform (SDP) Data & Integration team, supporting critical Defence and Aerospace programmes. This role is responsible for the design, implementation, governance and continuous improvement of enterprise-scale data platforms, pipelines and architectures. As a technical leader, you will drive the development of secure, scalable and resilient data solutions that enable data-driven decision-making, advanced analytics and digital transformation within highly regulated environments. Working closely with architects, engineering teams, business stakeholders and delivery functions, you will establish engineering best practices, modernise data platforms, and lead the adoption of DataOps and cloud-native technologies. Key Responsibilities Data Architecture & Engineering • Design and maintain secure, scalable and resilient enterprise data architectures and platforms. • Lead the development of enterprise data pipelines, integration frameworks and data products. • Establish and enforce data engineering standards, architectural patterns and best practices. • Drive the adoption of modern cloud-native data services and architectures. • Support data platform modernisation and cloud migration initiatives. • Ensure platform reliability, performance and scalability. DataOps & Automation • Implement DataOps and DevOps methodologies to improve deployment efficiency and operational reliability. • Develop and maintain CI/CD capabilities for data platforms and pipelines. • Automate testing, deployment, monitoring and recovery processes. • Promote Infrastructure as Code (IaC) and platform automation. • Improve engineering efficiency through standardisation and automation. Data Governance & Quality • Ensure data quality, integrity, lineage and consistency across enterprise platforms. • Establish monitoring and alerting for data quality and platform performance. • Support compliance with security, regulatory and information governance requirements. • Enable metadata management, auditability and data stewardship capabilities. • Drive adoption of data governance frameworks and standards. Strategic Leadership • Provide technical leadership, mentoring and coaching across data engineering teams. • Align data engineering initiatives with organisational and programme objectives. • Support delivery of advanced analytics, AI and machine learning capabilities. • Influence technology decisions, architecture roadmaps and investment priorities. • Build strong relationships with business and technical stakeholders. Planning & Delivery • Prioritise high-value initiatives and technical improvements. • Manage dependencies across multiple projects and programmes. • Provide technical oversight and delivery assurance. • Monitor and mitigate technical risks and delivery challenges. • Support effective resource planning and engineering capacity management. Skills & Experience Essential • Active SC Clearance. • Experience delivering data solutions within MOD, Defence Digital, Aerospace, National Security or similarly regulated environments. • Minimum 7 years' experience in Data Engineering, Data Platform Development or related disciplines. • Strong expertise designing and implementing enterprise-scale data architectures. • Extensive experience with cloud data platforms including Microsoft Azure, AWS or Google Cloud. • Strong proficiency in SQL, Python, Spark and modern data processing frameworks. • Experience building and managing ETL/ELT pipelines and streaming data solutions. • Practical experience with DataOps, DevOps, CI/CD pipelines and Infrastructure as Code. • Strong understanding of data modelling, data warehousing and master data management principles. • Experience with data governance, metadata management and data quality frameworks. • Excellent stakeholder engagement, communication and leadership skills. • Strong analytical, troubleshooting and problem-solving capabilities. • Experience working in Agile delivery environments. Desirable • Professional cloud certifications across Azure, AWS or GCP. • Experience with Azure Data Factory, Databricks, Microsoft Fabric, Snowflake, Apache Airflow or similar technologies. • Knowledge of AI/ML platforms and MLOps practices. • Experience with graph databases and knowledge engineering solutions. • Experience implementing Data Mesh or Data Fabric architectures. • Enterprise Architecture experience. • Experience leading large-scale data modernisation programmes. • Knowledge of Defence security standards and classified environments. • TOGAF, DAMA CDMP or equivalent industry-recognised certifications. What Success Looks Like The successful candidate will: • Deliver secure, scalable and resilient enterprise data platforms supporting Defence and Aerospace programmes. • Enable effective deployment of data, analytics and digital capabilities across SDP. • Drive automation, standardisation and continuous improvement initiatives. • Improve platform reliability, performance, governance and operational efficiency. • Support advanced analytics, AI and machine learning capabilities. • Act as a trusted technical leader across data engineering, platform and architecture domains.