Databricks Solution Architect

Role Summary We are looking for an experienced Databricks Solution Architect to design, architect, and implement enterprise-scale data and analytics solutions on the Databricks Lakehouse Platform. The ideal candidate will have strong expertise in Azure/AWS Databricks, Spark, Delta Lake, cloud data architecture, ETL/ELT modernization, and data governance. This role requires close collaboration with business stakeholders, data engineers, architects, and leadership to deliver scalable, secure, and high-performance data platforms. Key Responsibilities • Design end-to-end enterprise data platforms using Databricks Lakehouse architecture. • Lead solution architecture discussions with clients and recommend best practices for data modernization. • Build scalable ETL/ELT pipelines using Databricks, Spark, Delta Lake, and cloud-native services. • Design and optimize Medallion Architecture (Bronze, Silver, Gold) for enterprise data processing. • Architect batch and streaming data pipelines using Structured Streaming, Auto Loader, and Delta Live Tables. • Optimize Spark jobs, cluster configurations, partitioning strategies, and query performance. • Design data models for analytics, reporting, and machine learning workloads. • Implement CI/CD pipelines for Databricks using Azure DevOps, GitHub Actions, or Jenkins. • Establish data governance, security, Unity Catalog, lineage, and access control policies. • Work with cloud services including Azure, AWS, or GCP to build modern data platforms. • Collaborate with business teams to gather requirements and translate them into technical solutions. • Lead technical design reviews, architecture governance, and solution documentation. • Mentor development teams on Databricks best practices and coding standards. • Support production deployments, performance tuning, troubleshooting, and platform optimization. Required Skills • 10+ years of experience in Data Engineering and Data Architecture. • 4+ years of hands-on experience with Databricks. • Strong expertise in Apache Spark (PySpark/Scala/Spark SQL). • Experience with Delta Lake, Delta Live Tables, Unity Catalog, and Photon Engine. • Strong SQL programming and performance tuning skills. • Experience with Medallion Architecture and Lakehouse implementation. • Knowledge of Data Warehousing and dimensional modeling. • Experience with cloud platforms (Azure, AWS, or GCP). • Expertise in Azure Data Factory, AWS Glue, or equivalent integration tools. • Experience with Git, CI/CD pipelines, and Infrastructure as Code. • Strong understanding of security, governance, and data quality frameworks. Preferred Skills • Databricks Certified Data Engineer Professional or Databricks Certified Solution Architect. • Experience with MLflow, Feature Store, and MLOps. • Knowledge of AI/ML workloads on Databricks. • Experience with Kafka, Event Hubs, or other streaming technologies. • Familiarity with Power BI, Tableau, or other BI platforms. • Experience with Terraform or ARM/Bicep for infrastructure automation. • Knowledge of Unity Catalog governance and data lineage.