Data Engineer

This is W2 role. Responsibilities • The person on this role will play a crucial role in building scalable and cost-effective data pipelines, data lakes, and analytics systems. • Data Ingestion: Implement data ingestion processes to collect data from various sources, including databases, streaming data, and external APIs. • Data Transformation: Develop ETL (Extract, Transform, Load) processes to transform and cleanse raw data into a structured and usable format for analysis. • Data Storage: Manage and optimize data storage solutions, including Amazon S3, Redshift, and other AWS storage services. • Data Processing: Utilize AWS services like AWS Glue, Amazon EMR, and AWS Lambda to process and analyze large datasets. • Data Monitoring and Optimization: Continuously monitor and optimize data pipelines and infrastructure for performance, cost-efficiency, and scalability. • Integration: Collaborate with data scientists, analysts, and other stakeholders to integrate AWS-based solutions into data analytics and reporting platforms. • Documentation: Maintain thorough documentation of data engineering processes, data flows, and system configurations. • Scalability: Design AWS-based solutions that can scale to accommodate growing data volumes and changing business requirements. • Cost Management: Implement cost-effective solutions by optimizing resource usage and recommending cost-saving measures. • Troubleshooting: Diagnose and resolve AWS-related issues to minimize downtime and disruptions. Qualifications: Must Have: • Strong experience with AWS services on big data platforms. • Strong Python experience in building Dash application in AWS • Spark • Shell scripting and Linux knowledge Nice to have: • Investment Knowledge preferred • Looking for candidates who have worked in Investment/Finance space • Machine Learning Skills Qualifications: • Educational Background: A bachelor's degree in computer science, information technology, or a related field is typically required. • A master's degree is preferred with GPA of 3.8+ Problem-Solving: Strong analytical and problem-solving skills to address complex data engineering challenges.