Kafka Engineer

Job Title: Kafka Engineer Location: Pleasanton, CA (onsite) Job Description: Below is the JD, appreciate if you can share the profiles on priority. Kafka Confluent Data engineer requirement. Platform Design & Build • Design and operationalize a multi-environment Confluent Cloud footprint (Prod/Non-prod) including RBAC, Service Accounts, API Keys, Private Networking (VPC peering/Private Link), IP allowlists, and encryption. • Define topic architecture standards (naming, partitions, replication, retention, compaction), consumer group strategy, and exactly-once/at-least-once semantics. • Implement Schema Registry and Stream Governance (compatibility modes, lifecycle, breaking change controls). • Stand up connectivity via Confluent Cloud Connectors • Establish Terraform/IaC modules for repeatable provisioning (environments, clusters, topics, ACLs, connectors, schemas). Security, Compliance & Reliability • Partner with Security to implement least-privilege, RBAC, secrets management, and compliance (PII, PCI/SOX/GxP as applicable). • Define and enforce data governance (ownership, contracts, lineage, retention, DLP). • Build SLOs/SLAs, error budgets, and incident response runbooks; coordinate with SRE for DR, multi-zone resilience, and capacity planning. Observability & Operations • Instrument metrics, logs, traces across Confluent Cloud and clients (lag, consumer health, throughput, latency, error rates, dead letter queues). • Integrate platform telemetry with Datadog/Prometheus/Grafana/New Relic/Splunk; set alerts and automated remediation where feasible. • Manage upgrades, quotas, cost optimization, and platform hygiene (ACL audits, orphaned topics/connectors cleanup). Application Onboarding & Enablement • Partner with app teams to design event contracts, idempotent producers, retry & DLQ patterns, and backpressure handling. • Provide reference implementations and client libraries/configs (Java, .NET, Python, Node) with secure defaults and resiliency patterns. • Run an “onboarding desk”: intake, discovery, architecture review, capacity estimates, provisioning via Terraform pipelines, and go-live checklists. • Conduct training, documentation, and office hours; champion best practices for event-driven architectures.