Senior / Lead Data Engineer

Senior / Lead Data Engineer Location: London/Hybrid Reports to: Delivery Lead Level: Senior Manager / Lead Individual Contributor Contract: Contract – Outside IR35 Please note – Due to the number of applicants, only those who meet or exceed both the technical and business skills will be considered for the next phase. Only successful candidates will be contacted by our Talent team. Role Purpose We are seeking an exceptional Senior / Lead Data Engineer to design, build, and govern enterprise-grade data platforms that power analytics, AI, and operational decision-making. This role blends hands-on engineering, technical architecture, and stakeholder leadership. You will work across business and technology teams to translate complex requirements into scalable, secure, and high-quality data products. Key Responsibilities Data Engineering & Platform Delivery • Design, build, and maintain end-to-end data integration pipelines across cloud platforms (primarily Azure) using tools such as Azure Data Factory, Databricks, PySpark, SQL, and Python. • Develop scalable ELT/ETL pipelines to support analytics, AI/ML, BI, and operational systems. • Implement data quality monitoring, lineage, validation, and governance to ensure trusted data products. • Build and optimise data models for analytical and AI workloads (e.g., Azure SQL, Synapse, PostgreSQL, Databricks Delta). • Integrate data from complex ecosystems including APIs, geospatial services, SaaS platforms, on-prem systems, and third-party vendors. • Develop production-ready pipelines with CI/CD using Azure DevOps, Git, and release pipelines. Technical Architecture & Design • Own high-level and low-level technical designs across data platforms, APIs, and integration services. • Perform technical impact assessments for new data sources, features, and business requirements. • Define scalable and secure architectures considering performance, cost, resilience, and regulatory requirements. • Produce options papers, technical roadmaps, and cost-of-ownership models to support investment decisions. • Ensure operational readiness by defining service wrap, monitoring, incident management, and interface agreements with system owners. Analytics, AI & Business Enablement • Collaborate with data scientists, analysts, product owners, and business leaders to define KPIs and data products that drive measurable value. • Support feature engineering, model data pipelines, and AI use cases (e.g., demand forecasting, optimisation, clustering, ML pipelines). • Enable business intelligence platforms (e.g., Power BI, Tableau) with trusted, well-modelled datasets. Leadership & Stakeholder Management • Lead, coach, and mentor data engineers (including graduates and diverse teams). • Act as a trusted technical advisor to senior stakeholders and clients. • Translate business needs into technical solutions and clearly communicate risks, trade-offs, and recommendations. • Support sales, pre-sales, and roadmap definition for future data initiatives. Required Experience & Skills Experience • 10+ years in data engineering, analytics, or data architecture roles. • Proven delivery across enterprise-scale data platforms in regulated or complex environments (e.g., public sector, infrastructure, utilities, retail, FMCG). • Experience operating in consulting or matrixed organisations with multiple stakeholders. Technical Skills • Advanced: SQL, Python, PySpark • Cloud: Azure (ADF, Databricks, Synapse, Azure SQL, Blob Storage, Power Automate) • Data Platforms: Databricks, relational & analytical data stores, API integration • DevOps: Azure DevOps, CI/CD pipelines, source control • Analytics & AI: Data modelling, feature engineering, ML data pipelines • Integration: APIs, geospatial data, third-party systems, event-based pipelines Architecture & Governance • Data architecture and platform design • Data quality, lineage, and governance frameworks • Security, GDPR, and data compliance considerations Soft Skills • Exceptional stakeholder management and communication • Strong risk management and problem-solving • Proven team leadership and coaching • Ability to bridge business and technical teams What Success Looks Like • Trusted, scalable data platforms supporting AI, analytics, and operations • High-quality, governed, and auditable datasets • Strong adoption by business users • A high-performing, upskilled data engineering team • Clear technical roadmaps aligned to business strategy How do you apply? Contact us at talent@theproexgroup.com and include the following: • o Curriculum Vitae / Resume • o Current Employment Position (Including notice period) • o Contact Details • o Remuneration Expectation • o Optional Cover Letter Be sure to follow our LinkedIn profile for more exciting global opportunities across a range of industries and sectors