Lead Data Engineer

Job Title: Lead Azure Data Engineer Location: Pleasanton, CA(onsite) Duration : Long Term Contract Key Responsibilities: Lead the design and development of real-time and batch data pipelines using Azure Databricks, Apache Spark, and Structured Streaming. Build and optimize data ingestion and processing workflows using Kafka and Databricks streaming for high-throughput, low-latency applications. Write efficient Python code in Databricks Notebooks, integrating with various data sources and destinations. Work with DB2, MongoDB, and other enterprise-grade data systems to unify data sources and ensure high-quality analytics. Focus on performance tuning of Spark jobs, resource optimization, and cost-effective compute usage on Azure. Collaborate with platform and architecture teams to ensure secure, scalable, and maintainable cloud data infrastructure. Support CI/CD for Databricks pipelines and notebooks using tools like GitHub, Azure DevOps, and Infrastructure as Code. Mentor junior engineers and contribute to a high-performing, quality-driven engineering culture. Interface with product owners, data scientists, and business analysts to translate data requirements into production-ready pipelines. What You Bring Required Qualifications: 15+ years of experience in data engineering, with 7+ years in a technical leadership role. Deep hands-on experience with Azure Databricks and Apache Spark, including Structured Streaming. Proficiency in Python for building scalable and reusable data workflows. Experience with Kafka for real-time data ingestion and event streaming. Strong experience integrating with relational (DB2, PostgreSQL) and NoSQL (MongoDB) databases. Demonstrated expertise in performance tuning and scaling Spark jobs in production environments. Strong communication and analytical skills, with the ability to work cross-functionally and lead through influence. Preferred Qualifications: Experience with Java or Scala in Spark streaming or Kafka connector development. Familiarity with Azure services such as Data Lake, Data Factory, Synapse, and Event Hubs. Knowledge of CI/CD pipelines, GitHub workflows, and Infrastructure as Code (e.g., Terraform). Background in building data platforms in regulated or large-scale enterprise environments.  Lovepreet Singh Account Manager m: +1 669-309-1773 w: www.e-solutionsinc.com e: Love.s@e-solutionsinc.com LinkedIn: https://www.linkedin.com/in/singhlovepreet/ “Disclaimer: E-Solutions Inc. provides equal employment opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, gender, sexual orientation, gender identity or expression, national origin, age, disability, genetic information, marital status, amnesty, or status as a covered veteran in accordance with applicable federal, state and local laws. We especially invite women, minorities, veterans, and individuals with disabilities to apply. EEO/AA/M/F/Vet/Disability.”