Job Title: Azure Data Engineer Location: Remote ||SFO, CA preferred. Contract: Long term Job Summary: We are seeking a highly skilled Azure Data Engineer to join our data engineering team. The ideal candidate will have strong experience in designing, building, and maintaining scalable data pipelines and infrastructure on cloud platforms like Azure and AWS. You will be responsible for developing robust ETL processes, working with big data technologies such as Hadoop and Spark, and contributing to system design for real-time analytics and fault-tolerant architectures. Key Responsibilities: • Design, build, and maintain ETL/ELT pipelines using modern orchestration tools like Luigi, Azure Data Factory, or similar. • Develop scalable data lakes and data ingestion frameworks leveraging Azure Data Lake Storage, Databricks, or AWS S3. • Work closely with cross-functional teams to support real-time data processing and streaming analytics solutions. • Implement and optimize big data solutions using technologies like Hadoop, Apache Spark, and Delta Lake. • Design and implement secure, scalable cloud infrastructure using Azure (preferred) and AWS services for data storage and processing. • Ensure robust access control and security policies across cloud resources using RBAC, IAM, network security groups, etc. • Participate in system architecture discussions focused on fault tolerance, high availability, and performance optimization. • Automate infrastructure and deployments using CI/CD practices and Infrastructure as Code (IaC) tools (e.g., Terraform, ARM templates). • Monitor and troubleshoot pipeline and infrastructure issues to ensure data reliability and integrity. Required Qualifications: • Bachelor’s or Master’s degree in Computer Science, Information Systems, Engineering, or related field. • 4+ years of experience in data engineering, with at least 2+ years on Azure and/or AWS. • Strong programming skills in Python, Scala, or Java for building ETL workflows. • Hands-on experience with Luigi, Airflow, or other pipeline orchestration tools. • Expertise in Azure Data Services (Data Lake, Synapse, Data Factory, Databricks, Event Hubs). • Solid understanding of cloud access controls, identity management, and security best practices. • Deep knowledge of big data technologies – Hadoop, Spark, Hive, and related ecosystems. • Experience in real-time analytics and streaming data processing using Kafka, Spark Streaming, or similar. • Proven track record of contributing to the design of scalable, fault-tolerant architectures in cloud environments. Preferred Skills: • Azure certifications (e.g., Azure Data Engineer Associate, Azure Solutions Architect). • Experience with data modeling, data governance, and metadata management. • Familiarity with containerization (Docker, Kubernetes) and DevOps practices. • Exposure to machine learning pipelines or analytics platforms is a plus.