Senior Data Engineer

Senior Data Engineer – Azure Databricks (Hybrid, London) - Inside IR35 Contract. Location: London, UK (3 days in-office) Security Clearance: SC required Job Type: Contract We are looking for a highly skilled Senior Data Engineer to join a market-leading team working on a cutting-edge Azure Databricks platform for economic and financial data. This platform supports Monetary Analysis, Forecasting, and Modelling, making it critical to the organisation’s decision-making processes. You’ll work in a collaborative, fast-paced environment, contributing to architecture, optimising pipelines, and ensuring the highest data quality standards. What You’ll Do • Design, develop, and optimise data pipelines for large datasets from multiple sources (APIs, databases, financial data providers). • Implement complex data transformations using Spark (PySpark or Scala) and other tools. • Work extensively with Azure Databricks, including Unity Catalog, Delta Lake, and Spark SQL. • Ensure data quality, consistency, and governance, leveraging tools such as Azure Purview. • Integrate data from relational databases, streaming sources, and APIs. • Collaborate with economists, data scientists, and technical teams to translate requirements into scalable solutions. • Automate deployments and implement DevOps best practices, including CI/CD pipelines. Tech Stack & Skills • Azure Databricks (3+ years hands-on) • Python and Spark (PySpark) or Scala • SQL / NoSQL databases • Azure Data Factory, Blob Storage, Azure SQL Database • Data pipeline optimisation & architecture design • Streaming data: Kafka or Azure Event Hubs • Version control (Git) and Agile methodologies • DevOps tools: Azure DevOps, Jenkins, Docker, Kubernetes • Data visualisation tools: Tableau or Power BI • Azure certifications (e.g., Data Engineer Associate) a plus Why Join • Work on a critical data platform used for high-impact economic analysis. • Join a collaborative, skilled team where you can influence architecture and pipeline design. • Take ownership of complex, high-value projects. • Hybrid role: balance focused office collaboration with remote flexibility. Additional Details • Immediate start preferred; open to candidates with up to 2 weeks’ notice. • Contract role, daily rate with Inside IR35 required. • Active SC clearance only. And finally... Who are we? FORT - Future of Recruiting Technologists is a technical & IT search consultancy that specialises in placing experienced professionals & teams with leading technology companies in the North of England. We strive to provide our candidates with the most transformative opportunities available. For our clients we understand that technical expertise alone does not make for the best employee. We take time to ensure a strong cultural and technical fit through deep understanding of our client's business and project needs. Ultimately, we pair technologists with businesses based on more than just skills. Our values and standards are the bedrock of every interaction; Continual Development, Compassion and Consistency.