Data Engineer

Must Skills: 1. Spark 1. SQL 1. Hadoop Day to Day: • Coordinating with PM and Teammates to determine prioritization of existing and upcoming tasks. • Assist in creating, running and optimizing queries on Impala and Spark to extract data from the Data Warehouse. • Convert business requirements from client stakeholders into technical specs. • Build PySpark jobs based on the technical specs for multiple projects in parallel. • Monitor and Troubleshoot the execution of jobs. • Ingest Data into SQL Server and transform it for Custom Analytics models. • Generate Reports/ Dashboards to be then exported back to the client. • Brainstorm and collaborate with stakeholders to solve blockers, if any. • Ensure timely completion of daily deliverables. • Experience as a Data Engineer or in a similar role, with a strong understanding of data engineering concepts and methodologies. • Strong knowledge of writing and optimizing SQL queries to retrieve, manipulate, and analyze data efficiently. • Hands-on experience with big data technologies such as: o Apache Spark (PySpark, Spark SQL, Spark Streaming) o Hadoop ecosystem (HDFS/ Ozone, Hive, YARN) • Familiarity with ETL frameworks and the ability to design, implement, and maintain data pipelines. • Understanding data modeling concepts and database design to support scalable data solutions. • Familiarity with Python. • Ability to analyze and troubleshoot data issues and provide solutions with minimal supervision. • Basic knowledge of testing and validating data to ensure accuracy and consistency in data pipelines. • Excellent verbal and written communication skills, with the ability to articulate complex ideas clearly and concisely to both technical and non-technical stakeholders. • Bachelor's degree in quantitative discipline such as Engineering, Mathematics, Finance, Business, or a related field. Equivalent practical experience may also be considered.