Lead Data Engineer

About the Role We are seeking a Lead Data Engineer to join the Data & Insights team, focused on building scalable data platforms and delivering enterprise-level insights. This role will support the Enterprise Information Warehouse and play a key part in ETL development and cloud migration to Google Cloud Platform (GCP). You will work on designing and implementing high-performance data pipelines, partnering with cross-functional teams to solve complex data challenges and enable advanced analytics. What You’ll Do • Lead moderately complex data engineering initiatives aligned to enterprise strategy • Design, build, and maintain scalable, high-performance ETL pipelines • Develop and optimize data processing frameworks using Ab Initio, Spark, and GCP • Analyze business and technical requirements to design effective data solutions • Build and maintain data models, data warehouses, and analytics environments • Ensure data quality, reliability, and availability across platforms • Perform code reviews with an emphasis on testability and quality • Troubleshoot and resolve production issues in distributed data environments • Collaborate with engineers, analysts, and stakeholders to deliver data-driven solutions • Support cloud migration initiatives and modern data architecture strategies Minimum Qualifications • 5+ years of experience in Data Engineering or Database Engineering (or equivalent experience) • Strong hands-on experience with: • Ab Initio (ETL development) • Apache Spark (Core, SQL, DataFrames, Datasets) • SQL and Data Warehousing (Teradata, Oracle, PL/SQL) • 3+ years of experience with Python or similar scripting languages • Experience with Google Cloud Platform (GCP) or other cloud platforms (AWS/Azure) • Solid understanding of distributed systems and parallel data processing • Experience with workflow orchestration tools (e.g., Airflow, Autosys) • Proven ability to deliver end-to-end data engineering solutions • Experience working in Agile/Scrum environments • Strong verbal and written communication skills Preferred Qualifications • Prior experience in financial services • Leadership experience guiding teams or large-scale data initiatives • Experience with: • BigQuery, Hadoop, Hive • Kafka or real-time data streaming technologies • Non-relational databases (e.g., MongoDB) • Exposure to machine learning pipelines, Spark ML, or AI/LLM frameworks • Familiarity with CI/CD pipelines and infrastructure-as-code practices • Strong Unix/Linux and shell scripting skills Top Skills • Ab Initio • Apache Spark • Google Cloud Platform (GCP)