Senior Data Engineer (AI)

Job Title: Senior Data Engineer (AI)-HYBRID Job Type: Contract + extension Location: Pittsburgh, PA Job Duration: Contract through end of 2026 (extension eligible) !! NO C2C only W2 !! Note: Contract through end of 2026 (extension eligible) Please read the job description carefully Before APPLY! Job Description: We are seeking a Senior Data Engineer with strong expertise in modern data engineering and working knowledge of MLOps practices to support the Knowledge Navigator platform. This individual will play a critical role in building and maintaining scalable data pipelines that feed downstream analytics and AI/ML use cases. The ideal candidate is a hands-on engineer who thrives in Azure-based environments, is experienced with large-scale data processing using Databricks, and can operate effectively within established enterprise architectures. This role requires a strong focus on data reliability, pipeline performance, and adherence to enterprise data governance and security standards, while collaborating across engineering, analytics, and AI teams. Responsibilities • Design, build, and maintain scalable data ingestion and processing pipelines into Azure Data Lake Storage (ADLS) and Databricks • Implement batch and/or streaming data pipelines to support analytics and AI/ML workloads • Develop and optimize data transformations using Spark (PySpark) and SQL within Databricks • Integrate data from enterprise source systems (e.g., APIs, relational databases, SaaS platforms) into centralized data platforms • Ensure high data quality, integrity, and availability for downstream consumers • Monitor, troubleshoot, and optimize pipeline performance, reliability, and cost efficiency • Support pipeline automation, orchestration, and scheduling using tools such as Azure Data Factory or Synapse Pipelines • Collaborate with data scientists and ML engineers to support data and model pipeline workflows • Implement logging, monitoring, and alerting to proactively identify issues in production pipelines • Adhere to enterprise data governance, security, and access control policies • Work within established architectural frameworks while contributing improvements and best practices • Partner with cross-functional teams to ensure stable and scalable data operations Requirements- • 7+ years of professional experience (post-graduate) in data engineering or related field • Strong hands-on experience with Azure data services, including: • Azure Data Lake Storage (ADLS) • Azure Data Factory and/or Synapse Pipelines • Deep experience with Databricks, including Spark / PySpark development and performance optimization • Proven experience building and maintaining enterprise-scale data pipelines • Experience integrating data from diverse enterprise systems (databases, APIs, cloud platforms) • Solid understanding of data architecture principles, data modeling, and ETL/ELT patterns • Experience with pipeline orchestration, automation, and monitoring • Working knowledge of MLOps concepts, including supporting data workflows for model training and deployment • Strong troubleshooting and performance tuning skills in distributed environments • Familiarity with data governance, security, and access control frameworks • Proficiency in Python and SQL • Strong collaboration and communication skills Experience supporting AI/ML or GenAI platforms and workflows Familiarity with CI/CD pipelines for data engineering or ML workflows Experience with real-time/streaming data technologies (e.g., Kafka, Azure Event Hubs)