Principal Data Engineer - Data Architect II

Role Description Principal Data Engineer Nottingham/ London (Hybrid) Contract Inside IR35/ Permanent About The Role As a Principal Data Engineer, you will combine deep technical expertise with engineering leadership to drive the design, development and evolution of our cloud-scale data platform. You will provide technical leadership across distributed data processing, data products, streaming architectures and data platform capabilities, while remaining hands-on with coding, design and engineering best practices. You will play a key role in shaping engineering standards, mentoring teams, influencing technical direction and delivering robust, scalable data solutions that enable our Engineers, Data Scientists, Analysts and Governance teams to unlock business value. What You'll Do • Lead the design, development and continuous evolution of cloud-scale data platforms and data products across batch and streaming workloads • Act as the technical lead for complex data initiatives, providing guidance on solution design, engineering standards and implementation approaches • Design, build and optimise distributed data processing pipelines using PySpark, Python, Airflow and AWS services • Drive engineering excellence through code reviews, technical coaching, design reviews and adoption of software engineering best practices • Partner closely with Product, Architecture, Cyber Security, Data Science, Analytics and Governance teams to deliver scalable and reusable data capabilities • Champion modern data engineering patterns including Data Lakehouse architectures, ELT frameworks, Data Products and event-driven processing • Influence technology choices, engineering standards and platform roadmaps while collaborating with Architecture and Enterprise Technology teams • Improve platform reliability, scalability, observability and operational excellence through automation, monitoring and continuous improvement • Drive adoption of CI/CD, Infrastructure-as-Code, testing strategies and engineering quality standards across the Data Engineering function • Mentor and develop engineers, fostering a culture of technical excellence, continuous learning and innovation • Support the adoption of AI and ML platform capabilities by building trusted, scalable and governed data foundations • Contribute hands-on to the delivery of critical solutions, helping teams solve complex technical challenges and accelerate execution What You'll Bring • Significant experience as a Lead Data Engineer, Principal Data Engineer or Technical Lead within a large-scale, data-driven organisation • Strong hands-on software engineering expertise with Python, PySpark and Apache Airflow • Deep experience designing and building distributed data processing systems and large-scale data platforms on AWS • Strong knowledge of modern AWS data technologies including Glue, EMR, Lambda, DynamoDB, S3, EventBridge, Step Functions and related services • Proven experience building Data Lakehouse platforms, Data Products and ELT/ETL frameworks supporting analytics, ML and AI workloads • Expertise in streaming data architectures, event-driven systems and real-time data processing patterns • Strong understanding of data modelling, data quality, metadata management, governance and data platform best practices • Experience implementing software engineering best practices including CI/CD, automated testing, infrastructure-as-code and observability • Demonstrated ability to lead technical delivery across multiple teams and influence engineering direction without direct authority • Excellent stakeholder management and communication skills, with the ability to engage effectively across engineering, product, architecture and senior leadership audiences • Practical experience supporting Machine Learning and Generative AI platforms through scalable data engineering solutions • Passion for mentoring engineers and building high-performing technical teams • Strong problem-solving skills with the ability to balance strategic thinking and hands-on execution • Experience operating within regulated industries, ideally Financial Services Hurry & apply for a more detailed conversation! #UST Skills data architecture,data processing,aws glue,cloud technology,