Contract Principal Data Engineer (Databricks)

Role: Principal Data Engineer (Azure Databricks) Programme: Data Exchange Platform Industry: Commodities / Energy Trading Location: London (Hybrid) Contract: Initial 6-month contract (long-term transformation programme) Rate: £850-950/day (Inside IR35) X4 Technology are partnered with a global energy trading firm on a major transformation of its Data Exchange platform - the technical data backbone supporting physical commodities trading globally. The platform automates, reconciles and standardises transaction workflows across physical crude oil and refined petroleum products, providing a single, trusted source of data across the trading ecosystem. Joining as contract Principal Data Engineer, you'll play a key role in the continued development of the Data Exchange platform, combining deep Azure Databricks expertise with technical ownership across data architecture, streaming, integration and platform engineering. Responsibilities for the Principal Data Engineer (Databricks) • Own the end-to-end data and platform architecture, shaping scalable, secure and high-performance solutions across multiple workstreams • Lead the design and delivery of the Databricks data platform, covering batch and real-time ingestion, streaming, data modelling and integration across Azure • Set technical direction and engineering standards, driving best practice across performance, scalability, governance, security, testing and cost optimisation • Act as the technical focal point across Architects, Engineers, Programme Managers, Business Analysts and Trading stakeholders • Identify and implement opportunities for AI and intelligent automation, leveraging technologies including MLflow, Azure AI Foundry, Agent Bricks and Microsoft Copilot Requirements for the Principal Data Engineer (Databricks) • Extensive hands-on Azure Databricks expertise, with a strong track record designing and delivering enterprise-scale data platforms (PySpark, Python, SQL, Delta Lake, Structured Streaming, DLT) • Proven ownership of data platform architecture, with strong knowledge of data modelling, integration, governance, security, performance and cost optimisation • Experience leading complex, greenfield or large-scale data transformation programmes across multiple workstreams and engineering teams • Strong real-time data and integration experience, including streaming architectures and complex data flows (Kafka beneficial) • Excellent technical stakeholder management, with the ability to influence High-profile role at the heart of a global trading data ecosystem. You'll combine hands-on engineering with genuine technical influence, helping shape a highly secure, cloud-native data platform that connects trading desks, physical assets and external partners Initial 6-month contract with long-term programme scope, £850-950/day (inside IR35), hybrid in London. Interviews are being scheduled immediately, with a view to onboarding as soon as possible, so please apply now for immediate consideration.