Ab Initio Data Engineer

Role: Ab Initio Data Engineer Location: Berkeley Heights, NJ Work Mode: 5 Days Onsite Key Responsibilities: • Create complex Ab Initio continuous flow graphs, including stateful joins, sliding time windows, and aggregations. • Implement event-driven data pipelines using Kafka, MQ, and file streams. • Ensure the resilience of continuous flows, including checkpointing and recovery, to guarantee "exactly-once" processing. • Apply advanced Ab Initio components (e.g., Reformat, Rollup, Join, Partition) to ensure low-latency performance. • Proactively monitor live production streams to ensure 24/7 reliability and troubleshooting data issues • Develop ETL pipelines for batch and real-time data ingestion and transformation. • Implement and ensure data validation, data security, integrity, and compliance across big data platforms. • Monitor and troubleshoot performance issues in large-scale clusters. • Collaborate with data scientists, analysts, and application teams to deliver high-quality data solutions. • Automate workflows and improve operational efficiency using scripting and orchestration tools. Required Skills & Qualifications: • Deep understanding of credit card process system • Deep knowledge of GDE (Graphical Development Environment), EME (Enterprise Meta>Environment), Conduct>It, and Continuous Flows. • Understanding of Kafka, message queues, and real-time stateful services. • Proficiency in Unix/Linux shell scripting, SQL, and database technologies (e.g., Oracle, Teradata). • Experience in Java , Scala, Python or Kafka is plus. • Familiarity with Linux/Unix environments and shell scripting. • Understanding of data security, governance, and compliance standards. • Experience with cloud-based big data platforms • Exposure to containerization (Docker, Kubernetes) for big data workloads. • Knowledge of CI/CD pipelines for data engineering projects.