Azure Databricks DevOps Engineer

DevOps Engineer (Azure Data Services) About the Role We are looking for a skilled and innovative DevOps Engineer to join our clients growing team. You'll play a pivotal role in modernising our data operations by designing and implementing Continuous Integration and Continuous Deployment (CI/CD) pipelines for Azure Databricks and Azure Data Factory (ADF). This is a fantastic opportunity to shape the future of our Azure environment, driving automation, efficiency, and reliability across our data services. This role is remote, with very occassional abroad travel (paid for) and falls inside IR35. It will be an inital 6 month contract with a likelihood of extension. We have interview slots available. Key Responsibilities • Design, build, and maintain CI/CD pipelines for Azure Databricks and ADF • Collaborate with data engineers and developers to integrate code changes seamlessly • Automate deployment processes to reduce manual intervention and boost efficiency • Monitor and troubleshoot CI/CD pipelines, ensuring minimal downtime • Implement best practices for asset bundling and version control in Azure DevOps • Optimize Azure DevOps environments for scalability and performance • Ensure compliance with security standards across all deployment processes • Document CI/CD processes, procedures, and configurations What We're Looking For • Proven experience with Azure DevOps and CI/CD pipeline implementation • Proficiency in Databricks Asset Bundles (DABs) • Strong knowledge of Azure Databricks and Azure Data Factory • Hands-on experience with scripting/automation tools (PowerShell, Python) • Experience in selective deployment for ADF pipelines • Familiarity with containerization/orchestration (Docker, Kubernetes) is a plus • Excellent problem-solving and troubleshooting skills • Strong communication and collaboration abilities • Knowledge of CI/CD security best practices • Detail-oriented mindset with a focus on high-quality automation solutions • Bachelor's degree in Computer Science, IT, or related field • Minimum 5 years' experience in DevOps or related roles