Data Engineer

Data Engineer Whitehall Resources require a Data Engineer to work with a key client on a 6 month initial contract. • This role will involve on site work in Cambridge 2 days per week. • Inside IR35. Data Engineer Job Overview: We are seeking a motivated and detail-oriented Data Engineer with a passion for designing and delivering scalable, high-quality data solutions in Databricks. You will play a key role in building and evolving our enterprise data platform, creating trusted, AI-ready data products that power reporting, analytics, automation, and future AI capabilities across Enterprise IT. Using your technical expertise, you will design, build, and optimise modern data pipelines that integrate information from enterprise platforms such as ServiceNow, Jira, Azure DevOps, and other business systems. Working within a Lakehouse architecture, you will transform raw operational data into curated, reliable datasets that enable self-service analytics and business decision-making. You will thrive in a collaborative environment, working closely with Data Engineers, Analytics Developers, and business stakeholders to deliver scalable, well-governed data solutions while continuously improving engineering standards, automation, and platform capabilities. If you're passionate about building modern data platforms that become the foundation for analytics and AI, we'd love to hear from you. Responsibilities: • Design, build, and maintain scalable ETL/ELT pipelines using Databricks, PySpark, SQL, and Python. • Design and develop scalable data models using Data Vault modelling principles to support enterprise reporting, analytics, and AI-ready data products. • Build and maintain trusted data products using Lakehouse and Medallion Architecture (Bronze, Silver, Gold) principles. • Integrate data from enterprise platforms including ServiceNow, Jira, Azure DevOps (ADO), and other operational systems. • Optimise data pipelines for performance, scalability, reliability, and cost efficiency. • Implement automated data validation, testing, and quality controls to ensure trusted downstream reporting and analytics. • Design efficient schemas and data structures that support reporting, analytics, and future AI use cases. • Collaborate with data engineers, visualisation developers, and business stakeholders to translate business requirements into scalable data solutions. • Contribute to metadata management, data lineage, governance, and documentation to support trusted enterprise data. • Manage source code using Git and contribute to CI/CD pipelines and automated deployment processes. • Support continuous improvement of engineering standards, reusable frameworks, automation, and platform capabilities. Required Skills and Experience: • Minimum 4 years' experience designing and delivering data engineering solutions using Databricks. • Strong experience with Python, PySpark, and SQL for data transformation and pipeline development. • Experience designing and maintaining scalable ETL/ELT pipelines integrating data from multiple enterprise systems. • Strong understanding of Data Vault modelling methodologies (Hubs, Links, Satellites) and experience designing scalable enterprise data models. • Solid understanding of Delta Lake, Lakehouse architecture, and Medallion Architecture. • Experience implementing data quality frameworks, automated validation, testing, and monitoring. • Knowledge of data governance principles including metadata management, data lineage, and security. • Experience using Git and CI/CD practices for collaborative software development. • Strong analytical and problem-solving skills with a focus on building reliable, maintainable, and scalable solutions. Nice to Have Skills and Experience: • Experience building data pipelines that integrate data from Jira • Familiarity with Agile delivery methodologies. • Knowledge of AI/BI, Tableau, Power BI or other visualisation platforms. • Exposure to AI-ready data architectures, semantic modelling, or data products supporting machine learning or Generative AI initiatives. • Excellent communication and collaboration skills, with the ability to work effectively across technical and non-technical teams Familiarity with Agile delivery methods and iterative development practices! • Knowledge of data governance and data lineage documentation standards. • Exposure to automation and CI/CD frameworks within Databricks.