Senior Data Engineer (SC Cleared)

Senior Data Engineer Location: UK-based, hybrid working Security Clearance: Active SC clearance required OR BPSS Experience: Senior-level data engineering experience Working Style: Agile, product-led digital delivery We are looking for a Senior Data Engineer to lead the design, build and operation of scalable, secure and reliable data platforms for digital services. This role is aligned to DDaT standards and will suit an experienced Data Engineer with strong hands-on capability across data pipelines, SQL, cloud platforms, APIs, streaming, containerisation and data modelling. You will be responsible for building robust data solutions, supporting engineering best practice and working closely with multidisciplinary teams to deliver high-quality data products. Important: Candidates must already hold active SC clearance or be able to demonstrate current SC clearance status. Role Responsibilities You will design, build and maintain scalable batch and streaming data pipelines using technologies such as Kafka and cloud-native services. You will work with relational databases including AWS Aurora PostgreSQL, covering database design, SQL development, query optimisation and operational performance. You will support and improve containerised data workloads using Docker and deployments on Kubernetes platforms. You will design and implement logical and physical data models and contribute to robust data warehouse structures, including star and snowflake schemas. You will take ownership of data quality, validation, reconciliation, monitoring and reliability across key data assets. You will integrate data from internal and external systems through REST APIs, handling authentication, pagination, error handling, data transformation and ingestion patterns. You will collaborate closely with analysts, application teams, platform engineers, product teams and stakeholders to understand data requirements, API contracts, data formats and refresh patterns. You will also mentor junior engineers, contribute to engineering standards and support clear documentation across pipelines, schemas and operational processes. Essential Skills • Strong SQL skills, including complex joins, aggregations, subqueries and query optimisation • Strong understanding of ETL / ELT concepts and data pipeline design • Hands-on experience with Apache Kafka or similar streaming platforms • Strong knowledge of AWS data services and managed databases • Experience with relational databases, ideally AWS Aurora PostgreSQL • Experience working with Docker and Kubernetes environments • Strong understanding of REST APIs, HTTP methods, JSON payloads and API integration patterns • Experience ingesting data from internal and third-party APIs • Strong data modelling and data warehousing knowledge • Ability to troubleshoot data, pipeline, performance and reliability issues • Active SC clearance OR BPSS Desirable Skills • Experience working in DDaT-aligned digital delivery environments • Experience with Swagger / OpenAPI • Understanding of API security, including tokens, API keys and OAuth • Experience with Git and modern engineering workflows • Experience improving data quality, observability and monitoring • Experience mentoring junior engineers • Understanding of Agile delivery principles • Exposure to CI/CD and DevOps practices Experience Required • Strong commercial experience in data engineering, software engineering or cloud data delivery • Proven experience building and supporting production-grade data pipelines • Experience working in multidisciplinary agile teams • Experience delivering data solutions in secure or regulated environments would be highly beneficial This is a strong opportunity for an experienced Data Engineer to take ownership of modern data engineering delivery across cloud platforms, streaming, APIs, SQL, containers and data warehousing.