Data Engineering Technical Lead

Join a Fortune 500 company and lead transformative data initiatives that drive smarter trading decisions and support the energy transition. Work with cutting-edge technologies in a collaborative environment while shaping the future of enterprise data platforms. Primary Services is excited to announce the role of Data Engineering Technical Lead for our client. This position offers the opportunity to architect and deliver scalable, real-time data solutions using Azure and Databricks while collaborating with trading, analytics, and IT teams across the enterprise. As a Data Engineering Technical Lead, you will play a critical role in enabling data-driven innovation and ensuring high-quality, governed data pipelines that power strategic decision-making. Responsibilities: • Architect, design, and implement scalable Azure Data Lakehouse solutions using ADF, ADLS, Synapse, Stream Analytics, and Databricks. • Lead development of real-time data pipelines with Databricks DLT and Structured Streaming for high-volume data ingestion and transformation. • Optimize pipeline performance and cloud resource usage to ensure cost-effective operations. • Leverage Unity Catalog for metadata management, data lineage, and governance. • Integrate and manage data from ETRM systems to support Crude & Products trading analytics. • Collaborate with trading desks, analysts, and IT teams to deliver actionable insights. • Establish and enforce data governance frameworks ensuring compliance, security, and data quality. • Monitor pipeline performance and proactively resolve production issues. • Communicate complex technical concepts clearly to business and technical stakeholders. • Lead executive-level discussions and presentations to align data initiatives with strategic priorities. • Mentor and guide a high-performing team of data engineers and analysts, promoting continuous learning and capability development. Qualifications: • Bachelor’s or Master’s degree in IT, Computer Science, or related discipline. • 12+ years of experience in data engineering and analytics. • Deep expertise with Databricks DLT, Structured Streaming, PySpark, Python, and SQL. • Strong knowledge of Azure data services, including ADF, ADLS, Synapse, and Stream Analytics. • Proven experience optimizing Databricks for high-volume, real-time data processing. • Familiarity with ETRM platforms and trading data workflows. • Strong stakeholder management and executive communication skills. • Demonstrated leadership experience with cross-functional teams and strategic data initiatives. • Preferred certifications in Azure Data Engineering and Databricks. • Nice to have: experience with Kafka, Hadoop, Airflow, PowerBI/QlikSense, AI/ML, NLP, GenAI, Agile/Kanban, and CI/CD practices. Want to be notified of similar positions? Visit www.primaryservices.com/signup/ to sign up and have notifications delivered straight to your inbox!