Ray Inference Engineer

Position Summary: • Designing, implementing, and maintaining distributed systems to build world-class ML platforms/products at scale • Experiment with, deploy, and manage LLMs in a production context • Benchmark and optimize inference deployments for different workloads, e.g. online vs. batch vs. streaming workloads • Diagnose, fix, improve, and automate complex issues across the entire stack to ensure maximum uptime and performance • Design and extend services to improve functionality and reliability of the platform • Monitor system performance, optimize for cost and efficiency, and resolve any issues that arise • Build relationships with stakeholders across the organization to better understand internal customer needs and enhance our product better for end users Minimum Qualifications: • 5+ years of experience in distributed systems with deep knowledge in computer science fundamentals • Experience managing deployments of LLMs at scale • Experience with inference runtimes/engines, e.g. ONNXRT, TensorRT, vLLM, sglang • Experience with ML Training/Inference profiling and optimization for different workloads and tasks, e.g. online inference, batch inference, streaming inference • Experience with profiling ML models for different end use cases, e.g. RAG vs. code completion, etc. • Experience with containerization and orchestration technologies, such as Docker and Kubernetes. • Experience in delivering data and machine learning infrastructure in production environments • Experience configuring, deploying, and troubleshooting large scale production environments • Experience in designing, building, and maintaining scalable, highly available systems that prioritize ease of use • Experience with alerting, monitoring and remediation automation in a large-scale distributed environment • Extensive programming experience in Java, Python or Go • Strong collaboration and communication (verbal and written) skills • B.S., M.S., or Ph.D. in Computer Science, Computer Engineering, or equivalent practical experience Preferred Qualifications: • Understanding of the ML lifecycle and state of the art ML Infrastructure technologies • Familiarity with CUDA + kernel implementation • Experience with inference optimization and fine-tuning techniques (e.g. pruning, distilling, quantization) • Experience with deploying + optimizing ML models on heterogenous hardware, e.g. GPUs, TPUs, Inferentia, etc. • Experience with GPU and other type of HPC infrastructure • Experience with training framework like PyTorch, Tensorflow, JAX • Deep understanding of Ray and KubeRay