Data Engineer (Pyspark)

Senior Data Engineer (Developer) – Pyspark We are supporting a leading global financial markets infrastructure and data provider as they modernise and scale their core data engineering capabilities. This role sits at the centre of their transformation programme, delivering high-quality data pipelines, models, and platforms that underpin critical services across the business. Key Responsibilities Design, build, and optimise scalable data pipelines for both batch and streaming workloads: • Develop dataflows and semantic models aligned to analytics and reporting needs • Implement complex transformations and performance-focused data processing logic • Apply data validation, cleansing, and profiling techniques to ensure accuracy and consistency • Implement access controls, data masking, and compliance-aligned security protocols • Tune workloads and optimise performance across Spark, Fabric, and Azure components • Translate business requirements into technical solutions through close collaboration with analysts and stakeholders • Maintain clear documentation and contribute to internal knowledge repositories Essential Skills Strong experience developing within Microsoft Azure and Microsoft Fabric: • Proficiency in Spark programming including DataFrames, RDDs, and Spark SQL • Python and PySpark development experience, including notebook-based workflows • Hands-on experience with Spark streaming and batch processing • Delta table optimisation and Fabric Spark job development • Solid Java programming and OOP understanding • Experience working with relational and NoSQL databases • Familiarity with GitLab, unit testing, and CI/CD pipelines • Strong troubleshooting ability and experience working in Agile environments • Excellent communication skills with stakeholder-facing experience • Practical experience building ETL workflows, lakehouse architectures, dataflows, and semantic models • Exposure to time-series data, financial market feeds, transactional records, and risk-related datasets