PySpark Data Engineer (SC Cleared)

Job Title: PySpark Data Engineer Location: Remote (ad hoc travel to London office) Contract Length: 3 Months Start Date: ASAP IR35: Inside Interview Process: 1 Stage, MS Teams Clearance: SC We are supporting a public sector client in hiring a PySpark Data Engineer with strong Python, Delta Lake and Azure cloud experience to join a data engineering team delivering scalable, test-driven data processing solutions in a containerised environment. This is a hands-on engineering role requiring deep Python and PySpark proficiency, with a strong emphasis on behaviour-driven development, modular code design and Delta Lake optimisation. The successful candidate will work closely with cloud architects, data scientists and DevOps teams to deliver robust, well-tested pipelines across Azure infrastructure. This role suits an experienced Python Data Engineer comfortable with test-driven and configuration-driven development, confident working independently in a fast-paced Agile environment and familiar with Azure cloud services across Functions, Key Vault, Blob Storage and Data Lake Storage Gen2. Key Responsibilities • Develop and maintain data ingestion, transformation and validation pipelines using Python and PySpark • Implement unit and behaviour-driven tests using Behave, including robust mocking and patching of dependencies • Design and maintain Delta Lake tables for optimised query performance, ACID compliance and incremental data loads • Build and manage containerised environments using Docker for consistent development, testing and deployment workflows • Develop configurable, parameter-driven codebases to support modular and reusable data solutions across environments • Integrate Azure services including Azure Functions, Azure Key Vault, Azure Blob Storage and Data Lake Storage Gen2 • Collaborate with cloud architects, data scientists and DevOps teams to ensure seamless CI/CD workflows, version control and environment consistency • Troubleshoot and optimise Spark jobs for performance and scalability in production environments • Maintain technical documentation and adhere to best practices in cloud security and data governance Essential Skills • Strong Python proficiency with emphasis on modular, test-driven and configuration-driven design • Demonstrated experience writing unit tests and BDD scenarios using Behave or similar frameworks • In-depth understanding of mocking, patching and dependency injection in Python testing • Proficiency in PySpark with hands-on experience in distributed data processing and performance tuning • Solid understanding of Delta Lake concepts, transactional guarantees and schema evolution • Experience with Docker for development, testing and deployment workflows • Familiarity with Azure services including Azure Functions, Key Vault, Blob Storage and Data Lake Storage Gen2 • Ability to implement configuration-driven applications for flexible deployment across environments Desirable Skills • Experience developing in Databricks or Synapse with Delta Lake integration • CI/CD pipeline experience using Azure DevOps or similar tooling • Knowledge of data security and governance best practices within Azure ecosystems • Experience collaborating with distributed teams in Agile delivery environments