Ab Initio Data Engineer

Title: Data Engineer (Ab Initio, OzoneCH, Flink)/Ab initio developer Role: Streaming Data Engineer – Ab Initio, OzoneCH, Flink Location: Berkeley Heights, NJ (5 Days Onsite) Contract Job Description: We are seeking a highly skilled resource to design and implement high-performance, event-driven data pipelines, ensuring low-latency data processing and high availability system for the large credit card processing system. The ideal candidate will work with the Ab Initio ecosystem (GDE, EME, Conduct>It) to build stateful services that ingest, filter, and transform data from sources like Kafka or message queues, pushing updates to dashboards or downstream databases in near-real-time. Key Responsibilities: • Create complex Ab Initio continuous flow graphs, including stateful joins, sliding time windows, and aggregations. • Implement event-driven data pipelines using Kafka, MQ, and file streams. • Ensure the resilience of continuous flows, including checkpointing and recovery, to guarantee "exactly-once" processing. • Apply advanced Ab Initio components (e.g., Reformat, Rollup, Join, Partition) to ensure low-latency performance. • Proactively monitor live production streams to ensure 24/7 reliability and troubleshooting data issues • Develop ETL pipelines for batch and real-time data ingestion and transformation. • Implement and ensure data validation, data security, integrity, and compliance across big data platforms. • Monitor and troubleshoot performance issues in large-scale clusters. • Collaborate with data scientists, analysts, and application teams to deliver high-quality data solutions. • Automate workflows and improve operational efficiency using scripting and orchestration tools.