Senior Data Engineer – AWS & Big Data( Only W2)

Senior Data & Software Engineer Long Term Contract Remote Responsibilities • Design, document, build, automate and manage data ingestion pipelines for master data management, deep-learning, and predictive analytics • Build and maintain big data environments that are highly secure, scalable, flexible, and performant using appropriate SQL, NoSQL and NewSQL technologies. • Capacity to debug and troubleshoot legacy pipelines • Understand existing data models, backend data processes, platform architecture and inbound/outbound file processing • Collaborate with diverse technical and non-technical teams located across multiple time-zones on client deliverables and technical support • Ability to juggle between different time sensitive tasks • Participate-in and contribute to design and code review Qualifications • 7+ years experience in building, maintaining and migrating data pipelines in AWS stack including but not limited to EMR, Airflow, Redshift, S3 • 7+ years of experience in SQL and Python. Ability to write and optimize complex SQL queries • Strong understanding of the object-oriented design and programming paradigm • Experience in data analytics product solutioning and architecting • Basic understanding of change control software; e.g., JIRA, Git • Strong organizational, problem-solving, and communication skills • Ability to work effectively with cross-functional teams Position Summary We are seeking an experienced Senior Data Engineer for a contract engagement (6-9 months) to lead a focused data engineering project. This role involves ingesting a new customer dataset, implementing comprehensive data quality evaluation frameworks, and preparing the data for entity resolution processes. Primary Objectives: Data Ingestion: Design and implement pipelines to ingest customer data from multiple source systems Data Quality Assessment: Build evaluation frameworks to profile, validate, and assess data quality Data Standardization: Normalize and standardize data formats (names, phone numbers, emails, addresses) Entity Resolution Preparation: Prepare clean, standardized datasets optimized for entity resolution and record matching processes Deliverables: Production-ready data ingestion pipelines Automated data quality validation framework Comprehensive data profiling and quality assessment reports Standardized datasets ready for entity resolution consumption Technical documentation and knowledge transfer materials Required Technical Skills Core Technologies (Must Have): Programming Languages: Python, SQL (advanced proficiency required) Big Data Processing: PySpark - hands-on experience with distributed data processing and optimization Cloud Platform: AWS services including: EMR (Elastic MapReduce) S3 (data lake storage and organization) Glue (ETL/ELT workflows) Orchestration: Apache Airflow for pipeline scheduling and workflow management