AWS Data Engineer

(10+ years of experience required) Role Overview We are building a next-generation data platform and are looking for an experienced Senior Data Engineer to help design, develop, and optimize large-scale data solutions. This role involves end-to-end data engineering, modern cloud-based development, and close collaboration with cross-functional stakeholders to deliver reliable, scalable, and high-quality data products. Key Responsibilities Design, develop, and maintain scalable, testable, and high-performance data pipelines using Python and Apache Spark. Orchestrate data workflows using cloud-native services such as AWS Glue, EMR Serverless, Lambda, and S3. Apply modern engineering practices including modular design, version control, CI/CD automation, and comprehensive testing. Support the design and implementation of lakehouse architectures leveraging table formats such as Apache Iceberg. Collaborate with business stakeholders to translate requirements into robust data engineering solutions. Build observability and monitoring into data workflows; implement data quality checks and validations. Participate in code reviews, pair programming, and architecture discussions to promote engineering excellence. Continuously expand domain knowledge and contribute insights relevant to data operations and analytics. What You’ll Bring Strong ability to write clean, maintainable Python code using best practices such as type hints, linting, and automated testing frameworks (e.g., pytest). Deep understanding of core data engineering concepts including ETL/ELT pipeline design, batch processing, schema evolution, and data modeling. Hands-on experience with Apache Spark or willingness and capability to learn large-scale distributed data processing. Familiarity with AWS data services such as S3, Glue, Lambda, and EMR. Ability to work closely with business and technical stakeholders and translate needs into actionable engineering tasks. Strong team collaboration skills, especially within Agile environments, emphasizing shared ownership and high transparency. Nice-to-Have Skills Experience with Apache Iceberg or similar lakehouse table formats (Delta Lake, Hudi). Practical exposure to CI/CD tools such as GitLab CI, GitHub Actions, or Jenkins. Familiarity with data quality frameworks such as Great Expectations or Deequ. Interest or background in financial markets, analytical datasets, or related business domains. Job Type: Fixed term contractContract length: 12 months Pay: £85,000.00-£90,000.00 per year Benefits: Sabbatical Sick pay Work Location: Hybrid remote in London EC1A