Senior Data Engineer

No third party recruiters please Must be able to work W2 Job Summary: We are seeking an experienced Data Engineer with 6–10 years of expertise in designing, building, and maintaining scalable, high-performance data pipelines and processing frameworks. The ideal candidate will have strong hands-on experience with orchestration using Apache Airflow and distributed data processing with Apache Spark (EMR). This role requires a solid understanding of big data architecture, data engineering best practices, and a commitment to delivering efficient, reliable, and maintainable data solutions that align with both business and technical requirements. \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ Roles and Responsibilities: • Design, build, and manage scalable and reliable data pipelines using Apache Airflow. • Develop and optimize large-scale data processing workflows using Apache Spark, including both batch and Structured Streaming. • Collaborate with data architects, analysts, and business stakeholders to translate data requirements into efficient engineering solutions. • Ensure the quality, performance, and security of data processes and systems. • Monitor, troubleshoot, and optimize data workflows and job executions. • Document solutions, workflows, and technical designs. \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ Qualifications Required: • Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field. • 6–10 years of experience in data engineering or related roles. • Strong experience with Apache Airflow for data orchestration and workflow management. • Proven expertise in building and tuning distributed data processing applications using Apache Spark (PySpark), including both Structured Streaming and Batch. • Experience with AWS cloud-based data ecosystems, particularly Athena and Redshift. • Proficient in SQL (Athena version) and experienced in working with large datasets from various sources (structured and unstructured). • Experience with data lakes, data warehouses, and batch/streaming data architectures. • Familiarity with CI/CD pipelines, version control, and DevOps practices in a data engineering context. • Strong problem-solving and communication skills, with the ability to work both independently and collaboratively. \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ Preferred Qualifications: • Familiarity with additional big data technologies (e.g., Hadoop, Kafka). • Experience working in Agile/Scrum environments. • Knowledge of data quality frameworks and validation engines. • Experience with data catalog tools.