Senior Databricks Data Engineer

We are seeking an experienced Senior Databricks Data Engineer to design, develop, and optimize enterprise-scale data engineering solutions. In this role, you will build and maintain scalable Databricks data pipelines, implement modern data architectures, and support enterprise analytics, reporting, and Sales Incentive Compensation (SIC) processing. You will collaborate with cross-functional business and technical teams to deliver high-quality, scalable data solutions while driving automation, performance optimization, and data integration initiatives. Key Responsibilities • Design, develop, and maintain scalable data pipelines using the Databricks platform. • Build and optimize Delta Lake tables, Delta Live Tables, Unity Catalog, notebooks, and Databricks Workflows. • Design and maintain enterprise ETL/ELT processes for data ingestion, transformation, and analytics. • Develop scalable data models supporting reporting, business intelligence, and operational analytics. • Architect and optimize Databricks environments, ensuring high availability, performance, and scalability. • Build and support data pipelines for Sales Incentive Compensation (SIC), including quota management, commission calculations, attainment tracking, and payout reporting. • Partner with Sales Operations, Finance, Business Intelligence, and IT teams to translate business requirements into technical solutions. • Integrate Databricks with cloud platforms (Azure, AWS, or GCP) and enterprise data sources. • Troubleshoot and optimize data pipelines, notebooks, and workflows for performance and reliability. • Implement best practices for data quality, governance, automation, and documentation. • Stay current with Databricks technologies, including Unity Catalog, Delta Live Tables, Photon, and platform enhancements. Required Qualifications • 7+ years of professional experience in Data Engineering. • 3+ years of hands-on experience designing and developing solutions using Databricks. • Strong experience with: • Databricks • Delta Lake • Delta Live Tables • Unity Catalog • Databricks Workflows • Strong programming skills in Python and/or Scala. • Advanced SQL development experience. • Hands-on experience with ETL/ELT development, data modeling, and enterprise data pipelines. • Experience implementing cloud-based data engineering solutions on Azure, AWS, or GCP. • Experience integrating with technologies such as Azure Data Factory, Kafka, Event Hubs, or similar platforms. • Strong analytical, troubleshooting, and problem-solving skills. • Excellent communication and stakeholder management skills. • Ability to work independently while collaborating effectively within cross-functional teams. Preferred Qualifications • Experience supporting Sales Incentive Compensation (SIC) solutions. • Knowledge of incentive compensation platforms such as: • Anaplan • Oracle ICM • Varicent • Experience with commission calculations, quota management, attainment tracking, and payout processing. • Databricks Certified Data Engineer Associate or Professional certification. • Experience documenting enterprise data architectures and technical solutions. • Experience building scalable enterprise data platforms and modern cloud data architectures.