Senior Data Engineer

Principal Data Engineer London (Hybrid) | Contract | Inside IR35 | Up to £650 per day Principal Data Engineer | Financial Services Transformation Programme We are supporting a major financial services transformation programme and are seeking an experienced Principal Data Engineer to play a pivotal role in the delivery of a modern cloud-based data platform. This role is ideal for a senior engineering professional who combines strong technical expertise with a consultative approach. You will work closely with business stakeholders, Data Architects, Product Owners, and engineering teams to shape solutions, provide technical direction, and remain actively involved in the hands-on delivery of critical data initiatives. You will be responsible for designing and building scalable data solutions, establishing engineering best practices, and driving the successful migration of data assets from legacy technologies to a modern cloud-based architecture. While this role carries significant influence across the programme, it is not purely strategic and requires someone who enjoys being deeply involved in the technical implementation. Key Responsibilities • Lead the design and delivery of scalable, cloud-native data solutions across multiple workstreams. • Work closely with business and technical stakeholders to understand requirements and translate them into practical data solutions. • Provide technical leadership and guidance to Data Engineers and delivery teams. • Remain hands-on with the design, development, testing, and deployment of data pipelines and data products. • Design and implement modern Lakehouse architectures using Databricks and AWS services. • Establish engineering standards, development practices, and data quality frameworks. • Collaborate with Data Architects to deliver target-state platform architecture and technical roadmaps. • Support and execute the migration of data assets from legacy platforms to modern cloud technologies. • Drive optimisation, performance tuning, and operational excellence across data platforms. • Contribute to architecture reviews, governance forums, and technical design discussions. • Mentor engineers and promote engineering excellence across the wider team. • Act as a trusted advisor to stakeholders, balancing business objectives with technical best practices. Essential Experience Technical Leadership & Delivery • Proven experience operating as a Principal Data Engineer, Lead Data Engineer, or Senior Data Engineer within large enterprise environments. • Experience delivering complex data transformation and cloud modernisation programmes. • Strong stakeholder management and communication skills. • Ability to influence technical direction while remaining actively involved in hands-on delivery. • Experience working across cross-functional teams including Architecture, Product, Engineering, and Business stakeholders. Data Engineering & Platforms • Databricks • Delta Lake • Unity Catalog • dbt • Spark / PySpark • Data pipeline development and optimisation • Lakehouse architecture • Data integration and ETL/ELT frameworks • Data platform design and implementation Cloud Technologies AWS • S3 • Glue • Airflow • IAM • Lambda (desirable) Data Modelling & Architecture • Data Vault • Kimball • Data warehousing principles and best practices • Dimensional modelling • Enterprise data architecture concepts Modernisation Experience Experience delivering migrations and modernisation initiatives involving technologies such as: • Teradata • Hadoop • Oracle • SAS Preferred Experience • Financial Services or Banking sector experience. • Experience working within regulated environments. • Large-scale cloud transformation programmes. • Agile delivery methodologies. • Experience working alongside Data Product Owners and Data Architects. • Exposure to data governance and metadata management frameworks. • CI/CD, Infrastructure as Code, and DevOps practices within data platforms. Skills & Attributes • Strong hands-on engineering capability. • Consultative approach with the ability to challenge and guide stakeholders. • Excellent communication and stakeholder engagement skills. • Ability to operate effectively across both strategic and technical discussions. • Strong problem-solving and analytical skills. • Delivery-focused mindset with a commitment to engineering quality. • Passion for mentoring and developing engineering capability. • Comfortable operating within fast-paced transformation programmes. Contract Details • Day Rate: Up to £650 per day • Contract Type: Inside IR35 • Duration: Initial contract through to December, with potential for extension • Location: London (Hybrid, subject to confirmation) • Start Date: ASAP / Short notice preferred Apply If you are an experienced Principal Data Engineer with expertise across Databricks, dbt, Spark, and AWS, and enjoy combining technical leadership with hands-on engineering delivery, we'd like to hear from you. Apply with your latest CV for immediate consideration.