Lead Data Engineer

About the Role We’re looking for a Lead Data Engineer to design, build, and scale modern data platforms using Databricks, PySpark, and SQL. In this role, you’ll take ownership of complex data pipelines, guide architectural decisions, and mentor a team of data engineers while partnering closely with analytics, product, and business stakeholders. Key Responsibilities • Lead the design and development of scalable, high-performance data pipelines using Databricks • Build and optimize ETL/ELT workflows using PySpark and SQL • Architect data solutions for batch and (if applicable) streaming workloads • Ensure data quality, reliability, performance, and governance best practices • Collaborate with data scientists, analysts, and business teams to enable analytics and ML use cases • Optimize Databricks jobs for cost, performance, and scalability • Review code, enforce engineering standards, and mentor junior engineers • Participate in technical design discussions and contribute to long-term data strategy Required Qualifications • 8+ years of experience in Data Engineering roles, with recent experience in a Lead or Senior capacity • Strong hands-on experience with Databricks (workspaces, notebooks, jobs, Delta Lake) • Advanced proficiency in PySpark for large-scale data processing • Strong SQL skills for data transformation, performance tuning, and analytics • Experience working with large datasets in cloud or distributed environments • Solid understanding of data modeling, partitioning, and performance optimization • Experience leading technical initiatives or mentoring engineers Nice to Have • Experience with cloud platforms (AWS, Azure, or GCP) • Familiarity with Delta Lake, data lakes, or lakehouse architectures • Exposure to streaming technologies (Kafka, Structured Streaming) • Experience supporting analytics, BI, or ML workloads • Knowledge of CI/CD, data observability, or orchestration tools (Airflow, ADF, etc.)