Lead Data Engineer

Lead Data Engineer (Contract) 5-Month Engagement | Extension Possible Location: Remote A technology consulting firm is seeking a Lead Data Engineer to join a major data and platform modernization program for a U.S. based financial lender that specializes in the agricultural industry. This is a hands-on leadership role where you’ll own the architecture and delivery of a modern Azure-based data platform, while managing a small team of engineers. The Opportunity You’ll lead the design and build of a production-grade Azure Databricks lakehouse, replacing legacy reporting and ETL systems, and delivering scalable, governed data pipelines across multiple workstreams. This is a player/coach role — ideal for someone who enjoys writing production code, setting engineering standards, and leading from the front. What You’ll Be Doing • Own end-to-end technical architecture across multiple data workstreams • Build and scale an Azure Databricks lakehouse (Delta Lake, Unity Catalog, medallion architecture) • Migrate legacy SSRS/SSIS logic into modern data pipelines • Design ingestion pipelines for external platform data (parquet-based) • Implement secure data-sharing frameworks for downstream partners • Establish best practices across CI/CD, testing, observability, and data quality • Lead code reviews, mentor engineers, and set engineering standards • Partner with stakeholders across security, infrastructure, and business teams • Leverage AI coding tools (e.g., Copilot, Cursor, etc.) to accelerate development responsibly What We’re Looking For • 7+ years in data engineering, including leadership experience • Strong expertise in Azure Databricks (Spark, PySpark, Delta Lake, Unity Catalog) • Experience modernizing legacy data systems (SSRS/SSIS) • Deep knowledge of the Azure ecosystem (ADLS, ADF, Key Vault, networking) • Advanced SQL and Python skills with strong software engineering fundamentals • Experience leading teams in an agile, client-facing environment • Familiarity with AI-assisted development tools and best practices Nice to Have • Background in financial services or lending platforms • Experience with core systems (e.g., Loan Management Systems or banking platforms) • Exposure to data-sharing frameworks or partner data exchange • Databricks certifications • Experience with infrastructure-as-code (Terraform, Bicep)