Principal Data Engineer

Principal Data Engineer (Python / AWS / Kafka) Location: 3 days onsite in Canary Wharf, London Rate: £775 per day (Inside IR35) Contract Length: 12 months Sector: Financial Services Overview We are seeking a Principal Data Engineer to join a large financial services organisation undergoing a major data platform transformation. This role sits at the intersection of architecture, engineering leadership, and hands-on development, shaping a modern, cloud-native data ecosystem. You will lead the design and delivery of highly scalable, real-time and batch data platforms built on Python, AWS and Kafka, working closely with architecture, trading, risk, and analytics teams. Key Responsibilities Technical Leadership • Own the design and evolution of the enterprise data platform • Define best practices, engineering standards, and data governance patterns • Provide technical leadership to multiple data engineering squads • Drive adoption of event-driven and streaming architectures Platform & Engineering • Design and build high-throughput, low-latency data pipelines • Implement robust real-time streaming solutions using Kafka • Develop and maintain scalable Python-based data services • Architect secure and resilient AWS cloud data infrastructure • Ensure solutions meet regulatory, security and audit requirements Stakeholder Engagement • Partner with senior stakeholders across Risk, Trading, Compliance and Data Science • Translate complex business requirements into scalable technical solutions • Mentor senior and mid-level engineers across the programme Required Skills & Experience Core Technical • Expert-level Python engineering in data-intensive environments • Deep experience building streaming platforms using Apache Kafka • Strong hands-on experience with AWS, including: • S3, Lambda, ECS/EKS • Glue / EMR / Athena / Redshift • IAM, networking and security best practices • Experience building real-time and batch ETL/ELT pipelines • Strong experience with data modelling and distributed systems Architecture & Leadership • Proven experience operating at Principal / Lead Data Engineer level • Experience designing enterprise-scale data platforms • Strong understanding of event-driven and microservices architectures • Experience working in regulated financial services environments Desirable Experience • Experience with data mesh / data lakehouse architectures • Knowledge of CI/CD and DevOps tooling (Terraform, Docker, Kubernetes) • Experience with data governance and lineage tooling • Exposure to market, risk, or trading data environments Working Pattern • Hybrid: 3 days onsite in Canary Wharf, 2 days remote • £775 per day inside IR35 • 12-month contract with strong likelihood of extension