Mid-Level Data Engineer

Job Summary For Mid-Level Data Engineer • Support the IAM Data Lake Engineering initiative. • Build and maintain Data Lake solutions on Google Cloud Platform (GCP) using big data tools and technologies. • Design, develop, and optimize data pipelines for ingestion, processing, and transformation of large-scale datasets. • Implement data modeling and data processing solutions to support analytical and operational requirements. • Utilize PySpark for distributed data processing and transformation tasks. • Work extensively with the Hadoop ecosystem, including HDFS for data storage and management. • Apply deep knowledge of GCP architecture, including bucket structuring, naming conventions, lifecycle policies, and access controls. • Integrate and manage data using Neo4j for graph database requirements. • Ensure robust CI/CD processes for continuous integration and deployment of data engineering solutions. • Build both batch and streaming data ingestion pipelines leveraging GCP-native services. • Develop and maintain data consumption and exposure layers via views, APIs, and curated analytical datasets. • Collaborate with cross-functional teams to ensure data solutions are scalable, secure, and aligned with organizational standards. • Duration of assignment is 12 months, with a possible extension based on project needs and performance.