Kafka Engineer

Role: Platform Engineer- Kafka with Linux Duration: 6 - 12 months Location: NYC, NY Job description: Core Platform Engineer with deep expertise in Apache Kafka and Linux systems engineering. The ideal candidate will be responsible for designing, deploying, maintaining, and scaling Kafka clusters in mission-critical environments, while also supporting the Linux-based infrastructure that forms the foundation of our real-time data platform Responsibilities: • Design, implement, and maintain highly available Kafka clusters in production environments. • Perform Kafka upgrades and patching ensuring minimal downtime and seamless transitions. • Administer security protocols and access controls to protect the platform. • Monitor cluster health and performance using our established monitoring stack (Prometheus, Grafana) • Troubleshoot and resolve cluster-related incidents such as broker outages, replication lag. • Develop and enhance automation scripts to streamline operational tasks and reduce manual intervention. • Assist with planning and managing cluster resources to support future growth. • Develop Kafka topics, partitions, ACLs, schema registry, and optimize throughput and reliability. • Manage a tune Kafka Connect, Kafka Streams, and MirrorMaker as needed for data pipelines. • Ensure platform reliability and performance on enterprise-grade Linux systems (RHEL, Ubuntu). • Automate Kafka provisioning, configuration, and monitoring using Infrastructure as Code (IaC) and configuration management tools (e.g., Ansible, Terraform). • Deep knowledge of Kafka ecosystem components like ZooKeeper, Kafka Connect, Schema Registry, MirrorMaker, KRaft (Kafka without ZooKeeper). • Build custom Kafka tooling and dashboards for producers and consumers, monitor lag and health metrics.