Senior Data Engineer

Senior Data Engineer London 3 days per week on-site 6 months (likely extension) £550 - £615 per day outside IR35 Primus is partnering with a leading Financial Services client who are embarking on a greenfield data transformation programme. Their current processes offer limited digital customer interaction, and the vision is to modernise these processes by: • Building a modern data platform in Databricks • Creating a single customer view across the organisation. • Enabling new client-facing digital services through real-time and batch data pipelines. You will join a growing team of engineers and architects, with strong autonomy and ownership. This is a high-value greenfield initiative for the business, directly impacting customer experience and long-term data strategy. Key Responsibilities: • Design and build scalable data pipelines and transformation logic in Databricks • Implement and maintain Delta Lake physical models and relational data models. • Contribute to design and coding standards, working closely with architects. • Develop and maintain Python packages and libraries to support engineering work. • Build and run automated testing frameworks (e.g. PyTest). • Support CI/CD pipelines and DevOps best practices. • Collaborate with BAs on source-to-target mapping and build new data model components. • Participate in Agile ceremonies (stand-ups, backlog refinement, etc.). Essential Skills: • PySpark and SparkSQL. • Strong knowledge of relational database modelling • Experience designing and implementing in Databricks (DBX notebooks, Delta Lakes). • Azure platform experience. • ADF or Synapse pipelines for orchestration. • Python development • Familiarity with CI/CD and DevOps principles. Desirable Skills • Data Vault 2.0. • Data Governance & Quality tools (e.g. Great Expectations, Collibra). • Terraform and Infrastructure as Code. • Event Hubs, Azure Functions. • Experience with DLT / Lakeflow Declarative Pipelines: • Financial Services background. To be considered, please click apply.