Data Engineer

Job Description: Overview We are seeking an experienced Databricks Data Engineer to design, build, and optimize scalable data solutions that support enterprise reporting, analytics, data integration, and Sales Incentive Compensation (SIC) processes. This role will play a key part in architecting and managing Databricks environments, developing high-performance data pipelines, and delivering reliable, auditable data products that drive business decision-making. If you are passionate about modern data platforms, cloud technologies, and building end-to-end data solutions, this is an excellent opportunity to make a significant impact. Responsibilities • Partner with business stakeholders to understand data requirements and translate them into scalable technical solutions. • Design, develop, and maintain Databricks-based data pipelines, data models, Delta Lake tables, notebooks, and workflows. • Own and guide Databricks architecture, ensuring new and existing pipelines are designed for scalability, performance, and integration. • Enhance and optimize Databricks workspaces, workflows, and data processes to improve automation, efficiency, and reliability. • Build and support data pipelines for Sales Incentive Compensation (SIC), including: • Quota management • Attainment calculations • Commission processing • Payout reporting • Collaborate with Sales Operations and Finance teams to translate compensation plan rules into accurate and auditable data models and transformations. • Perform testing, troubleshooting, and performance tuning of data pipelines and notebooks. • Provide technical support for Databricks solutions and resolve complex data-related issues. • Leverage Databricks capabilities such as Unity Catalog, Delta Live Tables, and Photon to improve data quality, governance, and performance. • Collaborate with cross-functional teams across Finance, IT, Analytics, and Business Intelligence to ensure data alignment and integration. • Create and maintain technical documentation, including data models, architecture diagrams, pipeline configurations, and development standards. Required Qualifications • 7+ years of experience in Data Engineering. • 3+ years of hands-on experience designing, developing, and managing Databricks solutions. • Strong expertise with: • Databricks • Delta Lake • Delta Live Tables (DLT) • Unity Catalog • Databricks Workflows • Advanced SQL skills and strong programming experience using Python and/or Scala. • Deep understanding of data engineering principles, including: • Data ingestion • ETL/ELT development • Data modeling • Pipeline orchestration • Experience supporting Sales Incentive Compensation (SIC) processes, including incentive plan implementation, quota allocation, attainment tracking, commission calculations, and payout processing. • Ability to interpret complex business rules and translate them into scalable, maintainable data transformation logic. • Experience integrating Databricks with cloud platforms (Azure, AWS, or GCP) and modern data services such as Azure Data Factory, Event Hubs, Kafka, or similar technologies. • Strong stakeholder management skills with the ability to influence decisions and drive solution-oriented discussions. • Excellent analytical, troubleshooting, and problem-solving abilities. • Strong verbal and written communication skills with experience working across technical and business teams. • Self-starter with the ability to work independently while thriving in a collaborative team environment. • Experience documenting complex data architectures, data flows, and engineering standards. • Ability to manage multiple priorities in a fast-paced environment. • Strong focus on data quality, performance optimization, and delivering a high-quality user experience. Preferred Qualifications • Experience with Sales Incentive Compensation platforms such as Anaplan, Oracle ICM, Varicent, or similar solutions. • Experience integrating incentive compensation systems with enterprise data platforms. • Databricks Certified Data Engineer Associate or Professional certification. • Experience building enterprise-scale, cloud-native data architectures and analytics solutions. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender, identity, national origin, disability, or protected veteran status.