Monitoring Specialist

Observability Engineer (GCP & OpenTelemetry Specialist) Remote 12+ months We are seeking an expert in Cloud Observability to lead the design, implementation, and optimization of our monitoring infrastructure using OpenTelemetry (OTel) Collectors. You will be responsible for bridging the gap between our GCP infrastructure and our monitoring backends, ensuring that our telemetry data is high-quality, cost-effective, and actionable. Core Responsibilities OTel Collector Architecture: Design and deploy scalable OTel Collector pipelines (as sidecars, DaemonSets, or standalone services) on GKE and Compute Engine. Infrastructure Monitoring: Configure OTel receivers (e.g., hostmetrics, kubeletstats) to capture critical GCP resource health data. Label Enrichment & Context: Implement the resourcedetection processor to automatically inject GCP-specific metadata (Project ID, Zone, Instance ID, etc.) into all telemetry signals. Pipeline Optimization: Utilize batch, memory\_limiter, and transform processors to manage data volume, prevent OOM issues, and reduce GCP Cloud Monitoring costs. Backend Integration: Configure the googlecloud exporter to seamlessly route metrics to Google Cloud Monitoring and Managed Service for Prometheus. Required Technical Expertise 1. GCP-Specific Monitoring Strategy • The ideal candidate knows that "collecting everything" is a recipe for high costs and low signal. You should have deep experience with: • High-Value Metrics: Identifying and prioritizing "Golden Signals" (Latency, Traffic, Errors, Saturation) across GCP services like Cloud Run, GKE, Pub/Sub, and Cloud SQL. • Quota & Limit Management: Understanding GCP's metric descriptor limits (10,000 per project) and avoiding "Cardinality Explosions" from OTel labels. • IAM & Security: Setting up the necessary roles/monitoring.metricWriter and roles/compute.viewerpermissions for the Collector. 1. Advanced OTel Configuration • Label Enrichment: Expert-level knowledge of using the Resource Detection Processor with the gcp detector to ensure metrics are correctly mapped to GCP Monitored Resources. • Processor Logic: Proficiency in the Transform Processor (OTTL) to rename metrics or drop unnecessary labels before they reach the backend. • Deployment Patterns: Experience managing OTel configurations via Helm, Terraform, or the OTel Operator for Kubernetes. Preferred Qualifications • Experience with Google Managed Service for Prometheus (GMP). • Strong understanding of the OTLP (OpenTelemetry Protocol) and how it maps to GCP's CUMULATIVE and GAUGEmetric types. • Ability to build dashboards in Cloud Monitoring that correlate infrastructure metrics with application-level traces.