Microsoft Databricks Engineer

Position Overview We are seeking a highly skilled Microsoft Databricks Engineer to design, build, and support scalable data pipelines and Lakehouse solutions using Databricks on cloud platforms, primarily Azure. This role focuses on developing reliable data products, enabling enterprise analytics, and supporting modern data architecture initiatives using Medallion Architecture and emerging Data Mesh concepts. The ideal candidate will have strong hands-on experience with Databricks, Spark, Python, SQL, cloud data platforms, and modern data engineering practices. Key Responsibilities • Design, develop, and maintain scalable data pipelines and ETL/ELT processes using Databricks and cloud-based technologies • Build and support Lakehouse architectures using Databricks and cloud data platforms • Translate business requirements into reliable datasets, data products, and analytics-ready solutions • Implement and maintain data ingestion, transformation, and orchestration workflows • Support enterprise data architecture initiatives, including Medallion Architecture (Bronze, Silver, Gold) and Data Mesh principles • Develop and optimize data models, database structures, and storage solutions across relational and NoSQL platforms • Implement data quality, governance, security, and compliance best practices throughout the data lifecycle • Support workflow orchestration and scheduling using tools such as Airflow or Luigi • Monitor, troubleshoot, and optimize data pipelines and platform performance • Collaborate with analysts, architects, developers, and business stakeholders to support data-driven initiatives • Create and maintain technical documentation, standards, and operational procedures Required Qualifications • Strong hands-on experience with Databricks and core data engineering technologies including SQL, Python, and Apache Spark • Experience designing, developing, and maintaining ETL/ELT pipelines and data integration solutions • Experience working with cloud data platforms, preferably Microsoft Azure • Strong understanding of data modeling, data architecture, and database design principles • Experience working with relational and NoSQL databases • Knowledge of data governance, data quality, security, and compliance best practices • Experience with workflow orchestration tools such as Airflow or Luigi • Understanding of Medallion Architecture and modern data platform design principles • Familiarity with centralized and decentralized data architectures, including Data Mesh concepts • Strong troubleshooting, analytical, communication, and documentation skills Preferred Qualifications • Experience with Snowflake, Amazon Redshift, BigQuery, or other cloud data warehouse platforms • Experience with business intelligence and visualization tools such as Power BI or Tableau • Experience with data platform performance tuning, optimization, and troubleshooting • Knowledge of machine learning platforms and data science workflows • Experience with real-time streaming technologies such as Kafka or Flink • Professional experience with Go (Golang) • Experience working in healthcare, finance, or other regulated industries