Data Engineer - Azure, Fabric, ETL, SQL, PySpark

Job Title: Data Engineer Role Overview The Data Engineer plays a key role in designing, building, and maintaining data pipelines and infrastructure that support the organization’s analytics and reporting needs. This position focuses on converting raw data into actionable insights by ensuring data quality, consistency, security, and accessibility. The Data Engineer will collaborate closely with business data leads, BI engineers, and other stakeholders to deliver reliable, high-performing data solutions. Essential Responsibilities Data Pipeline Development • Design and implement scalable, complex ETL (extract, transform, load) data pipelines from diverse sources such as databases, APIs, and files. • Develop automated data ingestion, mock data generation, and validation processes. • Optimize data pipelines for performance and efficiency. • Maintain source control best practices (e.g., branch protection, Git-flow). • Implement CI/CD pipelines, including build, test, and automated deployment (e.g., Jenkins, Travis CI, Azure DevOps). • Demonstrate strong analytical and programming skills. Database Design and Tuning • Ensure referential integrity and query optimization across systems. • Design and maintain scalable database solutions that support high-performance analytics. Data Infrastructure Management • Manage and maintain data warehouses, data lakes, and other storage solutions. • Ensure data security, privacy, and compliance with applicable regulations. • Monitor system performance and troubleshoot issues proactively. Data Modeling • Develop and maintain data models that accurately represent business processes and entities. • Support data governance initiatives by implementing and maintaining policies and standards. Data Quality Assurance • Establish and implement data quality checks, validation rules, and error-handling procedures. • Identify and resolve data inconsistencies or quality issues. Tool Selection and Implementation • Evaluate, recommend, and implement modern data engineering tools and technologies. • Maintain and manage data engineering platforms to ensure optimal performance. Collaboration and Agile Development • Partner with business data leads, project managers, and stakeholders to define data requirements. • Provide technical guidance and support for data-driven initiatives. • Participate in agile development processes, including sprint planning, daily standups, and backlog grooming. • Analyze business and technical requirements to develop documentation, designs, and test plans. • Demonstrate flexibility by learning new technologies and contributing across the full stack when needed. Qualifications • Preferred: Experience with Azure Fabric and Azure Cloud environments. • Strong experience with relational databases (e.g., Oracle, SQL Server) and NoSQL databases. • Advanced proficiency in SQL and experience with complex queries and optimization. • Skilled in Python and PySpark for data engineering and automation. • Proven experience in performance tuning of data pipelines and database queries. • Solid understanding of DevOps, microservices, and modern application design principles. • Strong analytical, quantitative, and problem-solving abilities. • Excellent written and verbal communication skills; able to convey complex ideas through documentation and design diagrams. • Education: High School Diploma or GED required; Bachelor’s degree preferred. • Other Requirements: Valid driver’s license.