Data Engineer

Key Responsibilities: Pipeline Design & Development • Design and build robust, reusable, parameter-driven ingestion and transformation pipelines • using Azure Data Factory, Synapse Pipelines, Data Bricks and/or Microsoft Fabric Data Factory. • Implement medallion architecture (Bronze / Silver / Gold) on Azure Data Lake Storage Gen2 using Delta Lake, Parquet, and structured streaming patterns. • Build performant ELT workflows that leverage pushdown to source systems (Synapse Dedicated SQL Pool, Azure SQL, Teradata) where appropriate. • Develop and optimize PySpark notebooks and jobs on Azure Databricks or Synapse Spark. Data Modeling & Warehousin • gDesign dimensional models (Kimball star/snowflake) and data vault patterns for analytics consumption • .Implement Slowly Changing Dimensions (Type 1/2/3), Change Data Capture, and late-arriving data patterns • .Tune distributed SQL workloads in Synapse Dedicated SQL Pool / Fabric Warehouse, including distribution keys, partitioning, and clustered column store indexes . Platform Engineering & DevO • psImplement CI/CD for data pipelines using Azure DevOps (YAML pipelines, ARM/Bicep/Terraform) across Dev / SIT / UAT / Prod environment • s.Instrument pipelines with robust logging, auditing, and monitoring using Azure Monitor, Log Analytics, and KQ • L.Enterprise Data Warehouse (EDW) ETL/Informatica Develop • erDefine and enforce coding standards, code review practices, branching strategies, and release managemen t. Migration & Modernizat • ionLead or contribute to legacy-to-cloud migrations — e.g., Informatica PowerCenter to Azure Data Factory, on-premises Teradata / Oracle / SQL Server to Synapse or Fabr • ic.Perform workload assessment, capacity planning, and cost modeling for target-state architectur • es.production incident response for critical pipelin es. Required Qualificati • ons:Deep hands-on expertise with Azure Data Factory: pipelines, datasets, linked services, triggers, parameterization, mapping data flows, and all three Integration Runtime types (Azure, Selfhosted, SS • IS).Strong Experience in Data Bricks and PySp • ark.Production experience with one or more of: Azure Synapse Analytics (Dedicated and Serverless SQL Pools, Spark Pools) OR Azure Databricks (Delta Lake, Unity Catalog) OR Microsoft Fa • bric(Warehouse, Lakehouse, OneLa • ke).Strong working knowledge of Azure Data Lake Storage Gen2 (hierarchical namespace, RBAC + ACLs, lifecycle management, securi • ty).Experience with Azure Key Vault, Azure AD / Entra ID (including managed identities and service principals), and private networking (VNet integration, private endpoin • ts).Monitoring and troubleshooting with Azure Monitor, Log Analytics, and • KQL.Advanced SQL — window functions, CTEs, query optimization, execution plan analysis, performance tun • ing.Strong Python for data engineering — pandas, PySpark, REST API integration, unit testing (pyte • st).Proficient in T-SQL; familiarity with Spark SQL, KQL, PowerShell, and Bash shell script ing. Preferred Qualificat • ions:5+ years of data warehouse development experi • ence.5+ years of data modeling experience using ERWIN or similar t • ools.2+ years of experience with Azure Data Factory and Snowf • lake.Medicaid Domain Knowledge is a plus