Lead PySpark Developer

Lead PySpark Developer Location: Remote (Office days in FCS for Client’s team – no client site travel required) Contract: Until 30 June 2026 We are seeking an experienced Lead PySpark Developer to design, develop, and optimise complex data processing solutions on AWS. This is a hands-on engineering role focused on modernising legacy SAS workflows and delivering scalable, production-ready PySpark data pipelines within a financial services environment. Role Overview You will lead the development and optimisation of PySpark-based ETL/ELT solutions, supporting large-scale SAS-to-PySpark migration initiatives. The role requires deep expertise in distributed data processing, Spark optimisation, and AWS data services, combined with strong engineering discipline and clean coding practices. Key Responsibilities Technical Delivery • Design, develop, and optimise complex PySpark ETL/ELT and data mart solutions • Convert legacy SAS code to PySpark using SAS2PY and manual refactoring • Refactor and stabilise legacy data workflows into modern AWS-based architectures • Ensure high standards of data quality, reliability, and performance AWS & Cloud Engineering • Build and deploy scalable data pipelines using AWS services (EMR, Glue, S3, Athena) • Optimise Spark workloads for performance and cost efficiency • Work within Git-based version control and CI/CD pipelines Data & Engineering Excellence • Apply clean coding standards, modular design, logging, parameterisation, and exception handling • Troubleshoot and optimise distributed Spark workloads (partitioning, tuning, execution plans) • Develop and execute ETL test cases and validation processes Required Skills Core (Role Profile Alignment) • P3 – PySpark (minimum 5+ years hands-on experience) • P3 – AWS data services (EMR, Glue, S3, Athena, IAM) • P1 – SAS (Base SAS, Macros, SAS DI Studio – legacy code understanding and migration) Technical Expertise • Strong Spark performance tuning and optimisation experience • Deep understanding of ETL/ELT, data warehousing, SCDs, dimensions and facts • Strong Python coding and refactoring capability • Experience with distributed computing and big data architectures • Proficiency with Git workflows and CI/CD practices Desirable Experience • Banking or financial services background • Experience on SAS modernisation or cloud migration programmes • Exposure to DevOps tooling and Agile/Scrum delivery This is an excellent opportunity to play a key role in a large-scale data transformation programme, delivering high-performance cloud-based solutions within a collaborative and forward-thinking engineering team.