Technical Lead

Technical Lead /Sr SW Engineer Engagement: Contract-to-Hire Location: San Jose CA 3-day Onsite Hybrid Technical Lead /Sr SW Engineer - Infrastructure Telemetry & Capacity Planning Platform (Contractor) Overview We are seeking an experienced Technical Lead contractor to deliver hands-on software development for our Infrastructure Telemetry and Capacity Planning platform. This engagement covers the collection, streaming, processing, and persistence of telemetry data from various sources that contains utilization of servers and GPUs across our fleet, enabling efficient querying and dashboarding for capacity planning. The ideal contractor hits the ground running, engages actively in architecture design discussions, and drives execution with minimal ramp time. Scope of Engagement The contractor will serve as an active technical contributor and platform specialist working closely with cross functional, focused on the design, development, and delivery of an end-to-end telemetry pipeline that turns raw server and GPU metrics into a reliable, queryable capacity planning backbone. Telemetry Pipeline Design & Architecture — Participate in end-to-end solution design for streaming telemetry ingestion from servers (utilization, memory, thermal, power, and workload metrics), contributing to data flow models, schema design, and error-handling strategies. Recommend appropriate patterns (streaming vs. batch, event-driven ingestion, EMR-based processing) based on data volume, cardinality, and latency requirements for capacity planning use cases. Development & Implementation — Develop production-grade ingestion and processing pipelines following team coding standards, peer review practices, and CI/CD workflows. Build streaming collectors/agents for server telemetry, implement processing jobs on EMR (Spark), and persist processed data in NoSQL stores like Trino-queryable lakehouse formats (e.g., Parquet). Write adequate test coverage and instrument pipelines with logging and monitoring for operational observability. Required Qualifications • 10+ years of software development experience, with demonstrated technical execution on complex, large-scale data or infrastructure platforms • Strong proficiency in Java and/or Python; hands-on experience with Spark and distributed data processing • Deep, hands-on experience with AWS EMR (or spark) for large-scale data processing pipelines • Experience with streaming data ingestion technologies (e.g., Kafka, Flink) for high-volume telemetry data. • Solid experience with NoSQL data stores (e.g., DynamoDB, MongoDB) OR lakehouse/query engines such as Trino, including schema and partitioning strategies for efficient querying at scale • Experience collecting and processing infrastructure telemetry using tools such as Prometheus, Grafana. • Proven expertise in cloud-native architecture — high availability, extensibility; AWS preferred • Familiarity with containerized deployment environments (Kubernetes, Docker) and CI/CD pipelines • Strong experience using AI-assisted development tools (e.g., Cursor, Windsurf, or equivalent) to accelerate delivery and maintain code quality • Strong analytical and problem-solving skills with the ability to operate independently in an ambiguous environment Preferred Qualifications: • Working knowledge of capacity planning concepts — utilization forecasting, resource trending, and translating telemetry into actionable capacity signals • Experience building dashboards and visualization layers for operational and reporting. • Experience with Agile development methodologies and DevOps practices