Databricks Data Engineer (Hybrid - NYC)

With roots dating back to the 1800s, our Customer is a global professional services firm HQ’d in New York City with businesses in insurance brokerage, risk management, reinsurance services, talent management, investment advisory, and management consulting. Today, they are a unique group of businesses with many enduring competitive advantages. Among them are the quality of their colleagues, deep client relationships, vast global footprint, depth of intellectual capital, and cohesive and collaborative culture. We are seeking a Databricks Data Engineer on a contract basis to support our Customer’s business needs. This role is hybrid (3 days on-site, 2 days remote/week) in NYC. As a Databricks Data Engineer, you will be responsible for designing, developing, and implementing data engineering solutions using the Databricks platform to support data ingestion, transformation, performance reporting, analytics, sales incentive compensation processing, and enterprise data pipeline processes. Responsibilities • Collaborate with business stakeholders to gather requirements and translate them into scalable data engineering solutions • Design, develop, and maintain Databricks-based data pipelines, data models, Delta Lake tables, notebooks, and dashboards • Define and maintain the overall Databricks architecture to support integrated, scalable, and optimized data solutions • Enhance and optimize existing Databricks workspaces, pipelines, and workflows • Design and implement data pipelines supporting Sales Incentive Compensation (SIC) processes, including quota management, attainment tracking, commission calculations, and payout reporting • Partner with Sales Operations and Finance teams to translate incentive compensation rules into accurate data models and transformations • Perform testing, debugging, and performance tuning of data pipelines and notebooks • Provide technical support and troubleshoot Databricks-based solutions • Evaluate and implement new Databricks features such as Unity Catalog, Delta Live Tables, Photon, and other platform enhancements • Integrate Databricks with cloud platforms and enterprise data sources • Collaborate with Finance, IT, Business Intelligence, and other cross-functional teams to deliver end-to-end data solutions • Document data architectures, pipeline configurations, technical specifications, and development processes Skills And Qualifications • 7+ years of experience in data engineering • Minimum of 3 years of hands-on experience building and supporting Databricks solutions • Extensive experience with Databricks components, including Delta Lake, Delta Live Tables, Unity Catalog, and Databricks Workflows • Strong proficiency in Python and/or Scala • Advanced SQL skills • Experience with data ingestion, ETL/ELT development, data modeling, and pipeline orchestration • Experience designing and supporting Sales Incentive Compensation (SIC) data solutions • Knowledge of incentive plan design, quota allocation, attainment tracking, commission calculations, and payout processing • Experience integrating Databricks with Azure, AWS, or Google Cloud Platform • Experience with Azure Data Factory, Event Hubs, Kafka, or similar data integration technologies • Strong stakeholder management skills with the ability to translate complex business requirements into scalable technical solutions • Strong analytical, troubleshooting, and problem-solving skills • Excellent written and verbal communication skills • Experience documenting technical architectures, pipelines, and data models • Ability to manage multiple priorities and work independently in a fast-paced environment • Passion for data quality, performance optimization, and scalable data architecture Preferred Qualifications • Experience with Sales Incentive Compensation platforms such as Anaplan, Oracle ICM, or Varicent • Experience integrating incentive compensation platforms with enterprise data engineering solutions • Databricks Certified Data Engineer Associate or Professional certification We offer a competitive salary range for this position. Most candidates who join our team are hired at the median of this range, ensuring fair and equitable compensation based on experience and qualifications. Contractor benefits are available through our 3rd Party Employer of Record (Available upon completion of waiting period for eligible engagements). Benefits include: Medical, Dental, Vision, 401k. An Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, or protected veteran status and will not be discriminated against on the basis of disability.