Databricks Developer

Databricks Engineer Description The corporate IT Services team with our client is seeking a highly skilled Databricks Engineer with advanced expertise in Databricks CI/CD pipelines and streaming table development. The ideal candidate will have hands-on experience implementing robust CI/CD processes within the Databricks environment, enabling efficient code deployment, automated testing, and streamlined release management. Key responsibilities include designing and optimizing streaming tables to support real-time analytics, ensuring seamless schema evolution to accommodate changing business requirements. Candidates should demonstrate proficiency in schema optimization techniques to maximize performance and reliability, as well as a deep understanding of data engineering best practices on the Databricks platform. RESPONSIBILITIES • Develop, maintain, and optimize Databricks CI/CD pipelines for automated deployment and testing • Evaluate, understand and assist in planning for compute and job cluster resources in Azure Databricks environments • Design and implement streaming tables with support for schema evolution and optimization • Collaborate with data architects and business stakeholders to understand data requirements and deliver scalable solutions • Troubleshoot and resolve issues related to table schemas, data ingestion, and streaming performance • Ensure data quality, security, and compliance throughout the engineering process • Databricks environmental change tracking EDUCATION & EXPERIENCE QUALIFICATIONS • Bachelor’s degree in Computer Science, Engineering, or equivalent experience in related field • 5+ years of experience in data engineering, with a focus on cloud data solutions KNOWLEDGE, SKILLS, OR ABILITIES • Proficiency in Azure cloud computing and data architecture design • Implementation of a data lake solution using Azure Databricks, integrating various data sources, and enabling analytics for business intelligence • Proven experience with Databricks, including CI/CD pipeline management and streaming table development • Expertise in schema evolution and optimization within streaming data architectures • Strong programming skills in Python and SQL • Familiarity with YAML a plus • Proficient with GIT • Excellent problem-solving and communication abilities