Senior Azure Data Engineer

About the job: Our client, the nation’s largest provider-driven healthcare performance improvement company, is looking for a Sr. Azure Data Engineer with solid, real-time hands-on experience in: • PySpark • Databricks (notebooks, jobs, workflows, External/Managed tables, SQL Warehouse, AI/ML) • Databricks Cluster optimization • ADLS (Blob Storage) • Azure Data Factory • Azure SQL • Azure App Services Title: Senior Azure Data Engineer Location: Downtown Chicago/60606 (3 days onsite Tues, Wed, Thurs) What you will be doing: Key Responsibilities • Design, build, and support end-to-end data pipelines (ingestion, transformation, validation, publishing). • Develop and optimize SQL and PySpark/Databricks transformations for large datasets. • Build production-grade Python components (reusable modules, logging, error handling, testing). • Create and maintain Azure Data Factory (ADF) pipelines (triggers, parameterization, monitoring, failure handling). • Work within Azure environments (ADLS Gen2, Azure SQL, resource groups, portal operations). • Provision and maintain Azure components using Pulumi (Infrastructure as Code). • Participate in code reviews, documentation, and operational support (triage + root cause analysis). Must-have Skills (Required) • 5+ years' experience as a data engineer • 3+ years doing ETL / ELT Concepts: Strong understanding of pipeline patterns, incremental loads, data validation, and troubleshooting. • 3+ years in SQL: Advanced querying (CTEs, views, joins, complex query logic) and performance tuning for transformations and validation. • 2+ years using Python: Production-quality development (modular code, testing, logging, integration with APIs/files, CICD, Unit Test/Integration test automation, Code Coverage). • PySpark: Distributed transformations and performance optimization (joins, partitions, debugging), CICD, Unit Test/Integration test automation, Code Coverage. • 1+ years using Azure Data Factory (ADF): Build/operate ADF pipelines, parameterization, triggers, monitoring, retry/error handling; integrate with Databricks/ADLS. • 1+ year using Databricks: Develop and operationalize notebooks/jobs/workflows; Delta Lake patterns; basic cluster/job configuration. • Azure Fundamentals + Pulumi: Hands-on with ADLS Gen2, Azure Portal, Storage Explorer, Resource Groups, Azure SQL, and familiarity integrating with Azure OpenAI. Able to use/maintain Pulumi scripts for provisioning and managing Azure resources across environments. Nice-to-have Skills • R: Ability to support/translate validation rules with SQL scripts and create data quality reports. • TypeScript: Useful for pulumi pipeline to create Azure components. • Java: Useful for integration with existing services/components. • .NET: Useful for integration with existing services/components. • Angular / Spring Boot: Minor troubleshooting or coordination with app teams.