Data Engineer

Lead/Senior Data Engineer - Large Cap Private Equity Fund - Data Transformation Contract - Long Term Engagement - c.£600-800 p/d London - Hybrid A leading private markets investment firm is embarking on a significant data transformation programme and is looking for an experienced Databricks Engineer to play a key role in the journey. The business currently relies on a number of highly complex Excel-driven processes that support critical investment and operational activities. As part of a wider modernisation initiative, they're looking to build a scalable, governed data platform capable of supporting future growth, reporting, and decision-making. This is not a simple migration project. The successful candidate will work closely with the business to understand existing data structures, identify underlying business logic, and translate legacy processes into a modern Databricks architecture. Responsibilities • Analyse and modernise complex Excel-based data processes. • Design scalable data models, schemas and data structures within Databricks. • Build robust ETL/ELT pipelines using Databricks, SQL and PySpark. • Develop and maintain Bronze, Silver and Gold data layers. • Implement data quality, governance and validation frameworks. • Work closely with business stakeholders to understand reporting and operational requirements. • Help build a scalable data platform to support ongoing business growth. Requirements • Strong hands-on Databricks experience. • Advanced SQL and Python/PySpark skills. • Strong data modelling and schema design expertise. • Experience delivering data transformation or modernisation programmes. • Experience working with complex business-critical datasets. • Strong stakeholder management and requirements gathering skills. Essential Previous Private Equity, Private Markets, Asset Management or Investment Management experience is required. Candidates must be able to demonstrate experience working with investment, fund, portfolio, deal or related financial datasets and understand the challenges associated with operating within highly data-driven investment environments.