Lead PySpark Engineer

Lead PySpark Engineer | £281 - £292 We're working with a Tier-1 global financial services powerhouse on this exciting opportunity. Step into a high-impact leadership role driving a massive cloud-native transformation. You will move beyond a simple "lift and shift" to architect and refactor legacy SAS analytics into high-performance, distributed PySpark pipelines on AWS for one of the world's most respected financial institutions. The Role • Lead the technical migration of complex SAS Base and Macro logic into modern, modular Python code using PySpark and SAS2PY. • Design and develop robust ETL/ELT pipelines and scalable Data Marts using AWS EMR, Glue, and S3. • Tackle performance tuning at scale by optimizing Spark execution, including partitioning, shuffling, and caching strategies for massive datasets. • Architect modern data models utilizing SCD Type 2, Fact/Dimension tables, and Star Schemas within a production-ready cloud environment. • Drive quality through technical leadership, implementing rigorous CI/CD workflows, unit testing, and automated data reconciliation frameworks. What You'll Need • Expert-level proficiency in PySpark and Python, with a deep understanding of Clean Code and SOLID principles. • Strong hands-on experience across the AWS Data stack, specifically EMR, Glue, S3, Athena, and Lambda. • Proven ability to read, debug, and translate legacy SAS (Base, Macros, or DI Studio) into modern distributed computing architectures. • Experience with DevOps tooling and modern workflows, including Git, Jenkins, GitLab CI, and Infrastructure as Code via Terraform. • A background in Financial Services or a similar highly regulated environment where "penny-perfect" data accuracy is non-negotiable. What's On Offer • Competitive daily rate of £281 - £292 in a high-stakes, Tier-1 banking environment. • Remote-friendly working model providing excellent work-life balance. • Long-term contract potential within a massive digital transformation program. • Opportunity to work with the latest AWS data engineering tools at an enterprise scale. Apply via Haystack today!