Data Specialist

Summary This is a hands-on DataOps / Data Quality Engineer role with a strong focus on building data validation frameworks and automated testing for Azure-based data platforms. The role also includes DataOps responsibilities, ensuring reliable, observable, and well-governed pipeline operations across Fabric Data Factory, Azure Data Factory and Synapse environments. Additionally, the engineer will take on Data Reliability Engineering (SRE) responsibilities. Key Responsibilities • Build, maintain, or leverage open-source data validation frameworks to ensure data accuracy, schema integrity, and quality across ingestion and transformation pipelines • Test and validate data pipelines and PySpark notebooks developed by Data Engineers, ensuring they meet quality, reliability, and validation standards • Defining and standardizing monitoring, logging, alerting, and KPIs/SLAs across data platform to enable consistent measurement of data reliability. • Identify and create Azure Monitor alert rules and develop KQL queries to extract metrics and logs from Azure Monitor/Log Analytics for reliability tracking and alerting. • Write SQL queries and PowerShell (or another scripting language) to automate the execution of validation routines, verify pipeline outputs, and support end-to-end data quality workflows • Collaborate with Data Engineering, Cloud, and Governance teams to embed standardized validation and reliability practices into their workflows • Document validation rules, testing processes, operational guidelines, and data reliability best practices to ensure consistency across teams What We’re Looking For • Strong background in data validation frameworks, automated testing, data verification logic, and quality enforcement • Automation Experience for data validations, reconciliations and generating alerts. • Experience with Azure Monitor, setting up Alert rules, building dashboards using data queried (KQL) from Log Analytics. • Experience with Fabric Data Factory, Azure Data Factory, Synapse pipelines, and PySpark notebooks • Hands-on experience calling REST/OData APIs for validating data. • Experience writing SQL and scripts for programmatically doing data validations and reconciliation across systems. • Strong understanding of the Azure ecosystem, including identity, network security, storage, and authentication models • Working experience with Azure DevOps and CI/CD • Strong debugging, incident resolution, and system reliability skills aligned to SRE • Ability to work independently while collaborating effectively across Data Engineering, Cloud, Analytics, and Governance teams Beneficial Experience • Experience in data space, with strong exposure to data testing, validations, and Data Reliability Engineering • Experience defining and tracking data quality KPIs, operational KPIs, and SLAs to measure data reliability and performance • Hands-on experience using Azure Monitor, Log Analytics, and writing KQL queries to collect monitoring data and define alert rules • Experience writing SQL and PowerShell (or another scripting language) to automate data validation, reconciliation, and rule execution • Exposure to data validation frameworks such as Great Expectations, Soda, or custom SQL/PySpark rule engines • Experience validating pipelines and PySpark notebooks developed by data engineering teams across Fabric Data Factory, Azure Data Factory, and Synapse • Experience defining and documenting validation rules, operational testing guidelines, and reliability processes for consistent team adoption