Data Engineer

Contract Data Engineer - Specialty Insurance Contract Length: 6 Months Location: Hybrid / London Rate: £475 - £500 per day IR35 Status: Outside IR35 Start Date: ASAP An experienced Data Engineer is required to take ownership of the build of a data pipeline to consolidate bordereaux reporting data into a standardised, reporting-ready structure. This is a hands-on delivery role sitting within the BI team, where you will act as the sole Data Engineer on the project, responsible for end-to-end pipeline development, design, and delivery. You will be building scalable Azure Databricks-based data pipelines, working independently to define and implement robust, production-ready solutions that support reporting and downstream analytics. Key Responsibilities • Take ownership of the end-to-end build of a scalable ETL pipeline in Azure Databricks to ingest and consolidate bordereaux reporting data • Design and implement data ingestion processes leveraging existing email-based ingestion tooling for multiple file formats from shared mailboxes • Develop Databricks notebooks, jobs, and pipelines to support data transformation and orchestration using a medallion architecture approach • Build and implement data validation, quality controls, logging, monitoring, and error handling to ensure reliable pipeline performance • Integrate solutions with Azure DevOps CI/CD pipelines across development, test, and production environment Key Requirements: • Strong hands-on experience as a Data Engineer (5+ years required) • Proven experience with Azure Databricks, including building ETL pipelines and working with Delta Lake / Lakehouse architecture • Strong proficiency in Python, PySpark, and Spark S • Knowledge of Unity Catalog and modern Databricks governance and platform feature • Experience implementing CI/CD pipelines using Azure DevOps • Experience supporting BI/reporting layers such as Power BI or similar tools • Insurance experience is advantageous Data & Application Notice We process personal data submitted as part of your application for recruitment purposes in line with UK GDPR. All applications are reviewed by a member of our team, we do not use automated decision-making in our selection process. Your application will be assessed against the requirements of this role. By applying to this advert, you consent to receiving future relevant roles and industry news and insights from us. To opt out of this, please contact: compliance@pioneer-search.com. For more information on how we use your data, including how long we retain it & your rights, please refer to our Privacy Policy: www.pioneer-search.com/privacy-policy.