Data Engineer

Role - Data Engineering expertise in AWS (Data Lake). It’s remote. No C2C only W2 Overview: Seeking a skilled Data Engineer to join a dynamic team responsible for building and maintaining a Data Lake. The team leverages AWS Redshift to consume and store data from various sources, making it available for both internal and external stakeholders. The core of the work revolves around establishing seamless system-to-system connections via ETL processes and transforming data files for integration into the Data Lake. The primary database is Redshift, supported by DynamoDB for configuration management and EMR as the processing framework. Additionally, Postgres is utilized as the database type, and Python scripts along with Lambda functions are key tools for data ingestion and transformation. The team operates in an Agile environment, following Scrum methodology with 2-week sprints. This role requires a comprehensive understanding of the full stack of technologies used, as team members are expected to contribute across all areas of the data management ecosystem. Responsibilities: • Develop and maintain ETL processes to load data into the Data Lake using Python, Lambda functions, and other AWS services • Design and implement data ingestion pipelines that handle various file types and formats, ensuring they are parsed and prepared for consumption • Work with AWS Redshift to manage data storage and retrieval, ensuring optimal performance and data integrity • Utilize EMR for data processing tasks and DynamoDB for configuration management • Collaborate with team members in a full-stack environment, contributing to all aspects of data engineering from ingestion to storage • Participate in Agile ceremonies, contributing to sprint planning, daily stand-ups, and retrospective meetings Required Skills: • 4+ years of experience in data engineering • 2+ years of experience working with AWS Redshift and EMR • Strong experience with AWS Lambda and Python for building data pipelines • Experience with Postgres databases • Knowledge of DynamoDB for configuration management • Strong understanding of ETL processes and data lake architecture • Familiarity with Agile/Scrum methodologies, particularly in a JIRA-managed environment.