Azure Big Data Engineer (W2)

Job role: Big Data Engineer – Azure / Databricks / OT Data Location: Remote Duration: 6+ Months contract to hire W2, no C2C Note: willing to travel periodically to Parsippany, NJ and/or Richmond, VA once in a month (Travel expenses covered) Project Overview: • Client is building out a Digital Transformation and Data Engineering team focused on creating an enterprise Unified Data Platform / Unified Data Layer (UDL). • Today, critical business and operational data resides across numerous systems, including SAP/ERP, CRM, SaaS applications, cloud platforms, plant Operational Technology (OT), Historian systems, and other manufacturing platforms. Operational reporting is similarly fragmented across these environments. • The goal is to bring this data together into a trusted, governed, scalable Azure data platform that can serve as a unified source for enterprise reporting, Power BI, analytics, AI/ML, automation, finance, supply chain, logistics, HSE, commercial operations, and plant operations. • The organization is a Microsoft/Azure shop and is currently implementing Azure Databricks, with Azure Data Factory serving as a core integration/orchestration platform. Snowflake may potentially be introduced in the future. What They're Looking For: • Client is seeking a hands-on Big Data Engineer who can build production-grade pipelines connecting traditional enterprise applications with manufacturing g and OT environments. • This person must be technically strong but also self-sufficient and highly communicative. The engineer will work directly with data teams, IT, plant/controls teams, business stakeholders, and other departments to understand source systems and bring their data into the unified platform. • The ideal candidate has worked in continuous or batch manufacturing or an adjacent industrial environment such as chemicals, pharmaceuticals, CPG, oil & gas, energy, or utilities. Key Responsibilities: • Design and build scalable batch, CDC, near-real-time, and streaming data pipelines. • Build pipelines using Azure Databricks, Azure Data Factory, PySpark/Spark, SQL, ADLS/OneLake, and related Azure services. • Integrate data from ERP, CRM, SaaS, SAP S/4HANA/DataSphere, APIs, manufacturing applications, and OT systems. • Ingest and contextualize plant/industrial data from Historian, SCADA, PLC, IoT/IIoT, telemetry, and time-series environments. • Work with industrial connectivity/protocol concepts such as OPC UA and MQTT, along with secure OT-to-IT integration patterns. • Develop landing → curated → semantic/consumption layers within the Unified Data Platform. • Implement data contracts, schema/versioning, SCD handling, partitioning, caching, clustering, and performance optimization. • Build dimensional and semantic models supporting Power BI datasets, APIs, analytics, AI/ML applications, and agents. • Partner with plant controls/OT teams around signal quality, security, change control, network boundaries, and downtime windows. • Implement data-quality checks for freshness, completeness, schema changes, drift, and validation. • Establish monitoring, lineage, alerting, troubleshooting procedures, and production runbooks. • Implement RBAC, Key Vault/secrets management, data classifications, retention, governance, and MDM standards. • Automate testing and deployment through Git-based CI/CD and structured dev/test/prod environments. • Monitor and optimize Databricks/Azure performance and cloud costs. • Create documentation, data dictionaries, technical specifications, runbooks, and knowledge-transfer materials. Required Qualifications: • 5+ years of production data engineering experience building enterprise-scale data pipelines. • Strong hands-on experience with Azure Databricks. • Strong hands-on experience with Azure Data Factory (ADF). • Advanced PySpark/Spark, Python, and SQL skills. • Experience with Delta Lake/Lakehouse or similar modern data architectures. • Experience developing batch, CDC, and/or streaming pipelines. • Strong understanding of Spark Structured Streaming and data-processing performance optimization. • Experience integrating multiple enterprise source systems through APIs, databases, files, events, and streaming technologies. • Experience with Azure technologies such as ADLS, Synapse/Fabric, Azure SQL, Event Hubs, Key Vault, Azure DevOps, or equivalent services. • Experience implementing data quality, testing, monitoring, observability, governance, and CI/CD. • Strong communication skills with the ability to independently work with technical and nontechnical stakeholders. Critical OT / Manufacturing Requirement: • Candidates need meaningful experience with OT, IoT, industrial streaming, or time-series data.