Lead Data Engineer (SC Cleared)

Lead Data Engineer (SC Cleared) • 3‑month rolling contract • Remote working (with occasional site visit to regional hub Sheffield/Croydon) • Active SC Clearance required Overview We are seeking an SC Cleared Lead Data Engineer to support a UK Central Government programme focused on building secure, auditable, and intelligence‑led data platforms. You will provide technical leadership across data engineering activities, setting standards for data modelling, schema design, and auditability, while working closely with architects, analysts, and delivery teams. This role is highly hands-on and suited to a senior engineer capable of translating complex raw data into structured, trustworthy, and compliant datasets aligned to the POLE (Person, Object, Location, Event) data model. Key Responsibilities • Lead the design and delivery of secure, auditable data pipelines within a government environment • Structure, normalise, and integrate raw data sources into a POLE-aligned data model • Design and implement schemas aligned to POLE entities, ensuring consistency, extensibility, and performance • Build and optimise temporal data models, supporting historical truth and time-based analysis • Implement permission enforcement mechanisms using permission identifiers to control access at data and query level • Develop data processing workloads using AWS services, including Flink and Glue • Write and maintain high-quality Java-based data processing code • Design, implement, and optimise queries in PostgreSQL (or equivalent relational databases) • Ensure data solutions meet audit, compliance, and information assurance requirements • Embed database-level audit logging, capturing: • Who accessed data • What data was accessed • When access occurred • Why access was permitted • Act as a technical authority, mentoring engineers and setting best practices across the team • Collaborate with security, architecture, and governance stakeholders to ensure platform compliance Essential Skills & Experience • Strong experience designing and implementing temporal databases • Proven expertise in POLE data modelling, including: • Structuring raw data into POLE • Designing schemas aligned to POLE entities • Experience implementing fine-grained permission enforcement using permission identifiers • Hands-on experience with AWS data services, particularly: • AWS Flink • AWS Glue • Strong Java development background in data-intensive systems • Advanced experience with PostgreSQL or equivalent relational databases • Deep understanding of: • Schema design • Query optimisation • Data normalisation strategies • Strong experience building auditable data systems, including: • Database-level audit logging • Secure access tracking and traceability • Ability to work independently in a remote, delivery-focused environment Desirable Experience • Experience working within UK Central Government or public sector programmes • Familiarity with data used in law enforcement, intelligence, or investigative contexts • Experience designing systems for high-assurance or regulated environments • Prior technical leadership or lead engineer experience on complex data platforms