Azure Developer-Databricks

About the Role: We are seeking a highly skilled Senior Data Engineer with deep expertise in Databricks and Python, and substantial experience implementing medallion architecture. The ideal candidate will play a key role in designing and maintaining scalable, secure, and reliable data platforms that support advanced analytics and business insights. Experience in cybersecurity and/or healthcare data environments is highly preferred. Key Responsibilities: • Design, implement, and optimize robust data pipelines and ETL processes using Databricks and Python. • Architect and manage data solutions leveraging the medallion architecture (Bronze, Silver, Gold layers) for scalable and efficient data processing. • Ensure data quality, integrity, and compliance with security and privacy regulations, particularly in healthcare or cybersecurity contexts. • Collaborate with data scientists, analysts, and cross-functional teams to deliver high-quality, secure data solutions. • Develop and maintain data lakehouse architectures, utilizing Delta Lake and Unity Catalog. • Monitor, troubleshoot, and optimize data workflows in production environments. • Contribute to the development and enforcement of data engineering standards, especially with respect to secure data handling. Required Qualifications: • Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field. • 5+ years of experience in data engineering roles, with a strong emphasis on Databricks and Python. • Proven track record implementing medallion architecture in cloud-based environments. • Advanced proficiency in SQL, data modeling, and ETL frameworks. • Excellent problem-solving, communication, and collaboration skills.