Lead Data Engineer

Lead Data Engineer - Hybrid in Chicago, IL Local candidates only. Any Visa but need on our W2 only H1B won't work but Transfer will work No relocations candidates. Main Skills: Spark Scala AWS – S3, Step Functions, Lambda’s Airflow Databricks TB-scale data environments Data Warehouse/Lakehouse – Medallion Architecture Privacy/Data Governance The role As a Lead Data Engineer in the Epsilon Attribution Product Development team, you will: • Lead the design, implementation, and optimisation of large-scale data processing solutions using Scala, Spark, SQL, and modern data platform technologies for a major workstream of the attribution platform. • Lead the design and operation of trusted data processing pipelines that handle advertiser, customer, and measurement datasets within secure cloud environments (AWS, GCP, Azure) and approved data-sharing ecosystems. • Collaborate with Product, Data Science, Security, Privacy, and Platform Engineering teams to deliver privacy-preserving attribution, measurement, forecasting, and analytics solutions. • Own the design and operation of highly scalable batch and streaming data workflows using orchestration frameworks and cloud-native data services. • Implement data classification, access controls, and privacy-preserving processing techniques to ensure sensitive datasets and identifiers are handled in accordance with security and compliance requirements. • Drive the design and operation of clean-room and trusted data-sharing environments within your workstream, ensuring only approved aggregated or privacy-protected outputs are made available for downstream consumption. • Build observability, monitoring, and operational tooling to ensure reliability, performance, and compliance of data processing platforms. • Troubleshoot complex data platform, performance, and pipeline issues across distributed systems. • Drive technical design, architecture decisions, and engineering best practices for a major workstream within the attribution platform, in partnership with Staff/Principal engineers. • Mentor mid-level and senior engineers, lead design and code reviews, and provide technical leadership across your workstream. • Continuously enhance Epsilon's attribution, measurement, forecasting, and privacy-preserving analytics capabilities within your area of ownership. • Strong written and verbal English communication skills are required. • Good understanding of Agile/SCRUM methodologies and experience working within cross-functional product development teams.