Data Engineer Lead

Employment Eligibility Statement Due to specific project and client requirements, this position is open to U.S. Citizens and U.S. Lawful Permanent Residents (Green Card holders) . Sponsorship is not available at this time. Danta Technologies evaluates all candidates in compliance with the Immigration and Nationality Act (INA) and EEOC guidelines . All hiring decisions are made without regard to race, color, religion, sex, gender identity, sexual orientation, national origin, age, disability, veteran status, or any other protected characteristic. Note: “Open to candidates residing anywhere in the United States” Join a leading client in the Semiconductor & Manufacturing industry and drive enterprise-scale data transformation initiatives! Core Focus Designing scalable data architectures, building ETL/ELT pipelines, working with big data tools, and ensuring data quality, governance, and performance. They want a senior-level Data Lead who can own the entire data platform on Azure—not just build pipelines, but design, lead, and optimize end-to-end data solutions. Mandatory Skills: Azure Data Factory (ADF), Azure Databricks & PySpark, Azure Synapse, Azure SQL, Python, and Spark SQL Experience Requirements • 10 + years of IT experience in the Development/ Architecture Role • 4+ years of Proven experience in Data Engineering and architecture roles.(with Azure Data Factory , Azure Databricks & PySpark, Azure Synapse and Azure SQL). Qualifications • Bachelor's or master's degree in computer science, Engineering, or a related field. Skill Requirements • Strong Experience with ETL/ELT tools like ADF, Informatica , Talend etc., and data warehousing technologies like Azure Synapse, Azure SQL, Amazon redshift , Snowflake , Google Big Query etc. • Strong hands-on experience with Azure Data Factory (ADF) for data orchestration (for building and managing pipelines), Azure Databricks for big data processing and analytics, and Apache Spark for distributed data processing). • Adept in with big data tools(Databricks , Spark etc..) • Experience with Power BI, Tableau/OBIEE etc. • Proficiency in PySpark, Python, and Spark SQL. • Experience with CI/CD pipelines for data platforms. • Solid understanding of data warehouse best practices, development standards, and methodologies.. • Strong expertise in SQL and relational data modeling. • Experience designing data lakes, data warehouses, or Lakehouse architectures. • Good understanding of data governance, metadata management, and security best practices. • Solid understanding of distributed data processing and performance tuning. • Experience with ETL/ELT patterns and best practices. • Strong analytical, problem-solving skills and ability to work in a fast-paced, dynamic environment. • Excellent communication and documentation skills. Key Responsibilities • Design and implement end-to-end data architecture for batch and real-time data processing to implement enterprise-grade data solutions using Azure and Microsoft Fabric. • Design, build, and optimize ETL/ELT pipelines for ingestion, transformation, and publishing using ADF, Spark, and Databricks. • Architect and optimize data pipelines using Azure Data Factory (ADF). • Develop and manage scalable data processing frameworks using Databricks and Apache Spark. • Develop efficient data transformation logic using PySpark and Spark SQL. • Build reusable, high-performance data models for analytics and reporting. • Develop and maintain data ingestion, transformation, and orchestration workflows. • Ensure data quality, consistency, security, and governance across platforms. • Define and enforce data engineering best practices, including CI/CD, versioning, testing, and monitoring. • Mentor and guide data engineers, fostering a culture of innovation and excellence. • Collaborate with data engineers, analysts, data scientists, and business stakeholders. • Evaluate and integrate new tools and technologies to enhance data capabilities. • Own the data lifecycle, from ingestion and transformation to consumption and visualization. • Support production deployments, monitoring, and troubleshooting of data pipelines. • Document data architecture, processes, and best practices. Nice To Have • DevOps & CI/CD: Azure DevOps, GitHub Actions, Jenkins • Real-time streaming (Azure Event Hubs, Kafka), Microsoft Fabric Notes:- All qualified applicants will receive consideration for employment without regard to race, color, religion, religious creed, sex, national origin, ancestry, age, physical or mental disability, medical condition, genetic information, military and veteran status, marital status, pregnancy, gender, gender expression, gender identity, sexual orientation, or any other characteristic protected by local law, regulation, or ordinance. Benefits: Danta offers a compensation package to all W2 employees that are competitive in the industry. It consists of competitive pay, the option to elect healthcare insurance (Dental, Medical, Vision), Major holidays and Paid sick leave as per state law. The rate/ Salary range is dependent on numerous factors including Qualification, Experience and Location.