Contract Senior Data Engineer (Databricks)

Role: Senior Data Engineer (Azure Databricks) Programme: Azure Databricks Platform Industry: Commodities / Energy Trading Location: London (Hybrid) Contract: Initial 6-month contract (long-term transformation programme) Rate: £850-900/day (Inside IR35) X4 Technology are partnered with a leading global energy trading organisation delivering a major technology transformation centred on the migration from a legacy ETRM platform to Endur and the modernisation of its enterprise data platform using Azure Databricks. As part of this multi-year programme, the organisation is migrating Oracle-based data platforms into Azure Databricks to build a scalable, cloud-native data platform supporting Endur, global reporting, analytics and risk management, while reducing critical data latency from 10–15 minutes to near real-time. They are looking for an experienced Senior Data Engineer to play a key role in designing and building the Azure Databricks platform. This role combines deep hands-on engineering with technical ownership, helping define engineering standards, optimise platform performance and shape the future architecture of a business-critical data platform. Responsibilities of the Senior Data Engineer (Azure Databricks) • Design, build and optimise scalable Azure Databricks data pipelines and platforms using PySpark, SQL, Delta Lake and Structured Streaming. • Lead the migration of legacy Oracle data platforms into Azure Databricks, delivering high-performance data solutions for trading, reporting and analytics. • Drive engineering best practices across performance, scalability, CI/CD, testing, governance and cloud cost optimisation. • Collaborate with architects, platform teams and business stakeholders to shape technical solutions, mentor engineers and influence platform direction. Requirements for the Senior Data Engineer (Azure Databricks) • Extensive hands-on experience delivering enterprise-scale Azure Databricks data platforms, with strong expertise in PySpark, Python, SQL, Delta Lake and Structured Streaming (Delta Live Tables desirable). • Strong understanding of modern cloud data architecture, data modelling, performance optimisation, governance and data engineering best practices. • Experience working on large-scale cloud migration or platform modernisation programmes, with the ability to influence technical direction and mentor engineering teams. • Experience within energy trading, commodities, capital markets or financial services is advantageous but not essential. This is an opportunity to work on one of the largest Azure Databricks transformations in the energy trading sector, helping build a modern cloud data platform that underpins trading, risk, reporting and analytics. You'll have significant influence over technical decisions while remaining deeply involved in hands-on engineering within a high-performing team. Interviews are being scheduled immediately, with a view to onboarding as soon as possible. Apply now for immediate consideration.