Data Engineer

Role:- Data Bricks Data Engineer 📍 Location: New York, NY – 1166 🏢 Work Model: Onsite 4 days/week ⏳ Duration: 6 Months 💼 Contract Type: Fulltime W2 Contract – No C2H 🕐 Hours: 8:00 AM – 5:00 PM 🚀 Start: ASAP 🎯 Experience: 7+ Years 👥 Interview: 2–3 Rounds About the Role We are looking for an experienced Databricks Data Engineer to join a new team and new project within a leading financial services environment. This role will focus on building scalable data engineering solutions using Databricks to support data ingestion, transformation, analytics, performance reporting, and Sales Incentive Compensation (SIC) processes. The ideal candidate is a strong team player who can work independently, partner with business stakeholders, and translate complex business requirements into reliable, scalable data solutions. Key Responsibilities • Design, develop, and maintain Databricks data pipelines, Delta Lake tables, data models, notebooks, and workflows. • Develop scalable ETL/ELT solutions for data ingestion, transformation, reporting, and analytics. • Support Sales Incentive Compensation (SIC) processes including quota management, attainment calculations, commission processing, and payout reporting. • Partner with Sales Operations and Finance teams to translate incentive plan rules into accurate and auditable data transformations. • Optimize Databricks architecture, workspaces, pipelines, performance, and data quality. • Work with technologies such as Delta Live Tables, Unity Catalog, Photon, and Databricks Workflows. • Perform testing, debugging, troubleshooting, and performance tuning of data pipelines. • Integrate Databricks with cloud platforms and enterprise data sources. • Collaborate with Finance, IT, Business Intelligence, and other cross-functional teams. • Document data architectures, technical specifications, data models, pipelines, and development processes. Required Qualifications • 7+ years of Data Engineering experience. • 3+ years of hands-on Databricks experience. • Strong experience with Databricks, Delta Lake, Delta Live Tables, Unity Catalog, and Databricks Workflows. • Strong SQL skills and proficiency in Python and/or Scala. • Strong understanding of ETL/ELT, data ingestion, data transformation, data modeling, and pipeline orchestration. • Experience with Sales Incentive Compensation (SIC), including incentive plans, quota allocation, attainment tracking, commission calculations, and payout processing. • Ability to understand complex business rules and translate them into scalable data pipelines. • Experience integrating Databricks with Azure, AWS, or GCP. • Experience with data integration technologies such as Azure Data Factory, Event Hubs, Kafka, or equivalent. • Strong analytical, problem-solving, stakeholder management, and communication skills. • Ability to work effectively in a team environment while also being an independent self-starter. Preferred Qualifications • Experience with SIC platforms such as Anaplan, Oracle ICM, or Varicent. • Databricks Certified Data Engineer Associate or Professional certification. • Experience with data quality, performance optimization, and end-to-end enterprise data solutions. • Strong experience documenting complex data architectures and pipelines. What We're Looking For We're looking for someone who is more than just a keyword match—a hands-on Databricks Data Engineer who understands both the technical and business sides of data, can challenge requirements constructively, and can build reliable solutions in a fast-paced financial services environment.