Data Architect

Primary Responsibilities: ETL Pipeline Design and Development: • Lead the design and development of high-performing ETL processes to integrate and transform data across disparate sources • Deliver efficient, reliable pipelines that meet business needs and maintain the highest standards of security • Utilize ADF to automate and streamline data workflows, ensuring smooth transitions from source to target Data Integration and Transformation: • Build and manage complex ETL workflows that extract, transform, and load data for downstream analytics and reporting, ensuring data is accurate, timely, and secure • Take ownership of data quality and validation, creating resilient ETL processes that ensure only trusted data reaches its destination Cloud Platform Expertise: • Leverage the full power of the Azure ecosystem—ADF, Databricks, Synapse, and Purview—to manage and process high volumes of structured and unstructured data, delivering solutions that are scalable and performance-optimized • Integrate large datasets into Azure Synapse Analytics, enabling analytics teams to deliver data-driven insights that support the Department’s mission Performance Optimization: • Continuously optimize ETL jobs to minimize latency and maximize throughput • Ensure the architecture supports fast, reliable data access for end-users and systems, meeting stringent performance metrics Security and Compliance: • Embed security and compliance best practices in every step of the ETL process • Protect sensitive data by adhering to industry standards and ensuring compliance with the Department’s data governance policies • Use Azure Purview to enforce data governance, track data lineage, and ensure that data handling meets the highest standards of integrity Collaboration and Stakeholder Engagement: • Partner with cross-functional teams, including data engineers, analysts, business stakeholders, and security experts, to design and implement ETL solutions that meet the Department’s evolving needs • Act as a technical leader and mentor, helping guide junior team members and providing expert guidance on data processing and transformation best practices Documentation and Best Practices: • Develop and maintain clear, detailed documentation for ETL processes, ensuring the team can consistently deliver high-quality, reliable solutions • Establish and enforce best practices for data handling, ETL development, and security, driving a culture of excellence and accountability Required Qualifications: • Seven (7) or more years of experience in ETL development and data engineering • Three (3) or more years of hands-on experience working with ADF, Azure Cloud, Azure Databricks, Azure Synapse Analytics, and Azure Purview • Proven track record of building and optimizing large-scale ETL pipelines for high-performance, high-availability environments • Extensive expertise in Spark, Python, and/or Scala for large-scale data transformations • Strong Structured Query Language (SQL) proficiency and experience working with complex data structures • In-depth knowledge of data governance, security protocols, and role-based access control (RBAC) within the Azure ecosystem • Ability to design ETL processes that are resilient, efficient, and fully compliant with regulatory standards Preferred Qualifications: • Possession of a Microsoft Office Certification as an Azure Data Engineer Associate, Azure Solutions Architect Expert, and Azure Fundamentals • Azure Databricks Certification as a Data Engineer Associate