Lead Data Engineer

Lead Data Engineer Rate: £700 to £800 a day IR35: Outside IR35 Contract: 12 months Location: Remote UK, with one engineering day per month near Tottenham Court Road, London Engagement: Ltd company contractors only. The client will not engage contractors through umbrella companies. We are looking for a hands-on Lead Data Engineer to establish and shape a new data engineering squad for a major consumer technology business. This is an opportunity for an experienced contract lead to take ownership of a genuinely greenfield and ambiguous problem space. The business works with a broad ecosystem of external partners across retail, travel, transport and consumer services, each with different systems, technical capabilities and approaches to exchanging data. The goal is to create a more consistent and scalable way of onboarding partners, ingesting and exchanging data, and supporting secure data-sharing capabilities between organisations. The contract The Data Exchange squad is being rebuilt and will initially be small. You will work closely with product, architecture and engineering leadership to define how the team should operate and what the platform should become. You will be responsible for: • Shaping the technical direction of the Data Exchange platform • Designing and building Python-based data services • Creating repeatable approaches to partner onboarding and integration • Designing scalable data ingestion and exchange capabilities • Supporting data-sharing use cases between external organisations • Exploring capabilities such as data clean rooms • Guiding engineers through architecture, design and delivery decisions • Working with product to turn loosely defined requirements into deliverable features • Remaining hands-on across technical discovery, proof-of-concepts and implementation • Establishing engineering standards without over-engineering the solution This is not a coordination-only leadership position. The squad will be lean, so you will need to contribute directly while acting as the first point of technical escalation for the engineers around you. What we are looking for • Strong commercial Python engineering experience • A strong background in data platform or platform engineering • Previous experience as a Lead Data Engineer, Tech Lead, Staff Engineer, Principal Engineer or hands-on Engineering Manager • Experience designing data ingestion, integration or data-sharing platforms • Comfortable working through ambiguity and shaping new engineering teams • Strong architecture and systems-design fundamentals • Able to work effectively with product managers and non-technical stakeholders • Experience guiding senior engineers while continuing to contribute hands-on • A pragmatic approach to standardisation, scalability and delivery • Databricks experience would be useful, but is not essential • Interest in AI-assisted engineering and agentic development workflows Experience using OpenSpec, specification-driven engineering or other agentic frameworks would be a significant advantage. The organisation is actively investing in these ways of working and already has internal playbooks in place. The interview will be a practical technical discussion with senior data leadership and is likely to run slightly longer than one hour. Please note that the client will only engage contractors operating through their own Ltd company. Applications from contractors requiring an umbrella company cannot be considered.