Lead Observability & Data Architect/Engineer (SC/DV Cleared)

We are hiring for Lead Observability & Data Architect/Engineer (SC/DV Cleared) Job Type : Open for Contract (Outside / Inside IR35) & Permanent hiring Location : Hybrid - London/Reading/Birmingham/Manchester/Glasgow • We are seeking an Enterprise Platform Architect or Senior Data Engineer to design and scale national-critical monitoring fabrics and data hubs. Whether your background is rooted in core Observability platforms (Opensearch/Elastic/Splunk/OpenTelemetry) or high-scale Data Engineering (Kafka, Python, Airflow, OpenSearch/Lucene), your core mission is the same: transforming billions of raw event logs into actionable, secure streaming telemetry. • Should be able to build high-performance data pipelines to ingest, parse, and structure massive event telemetry across hybrid-cloud and air-gapped environments. • Manage large-scale cluster deployments, designing optimized indexing, data lifecycle policies, and schema mapping (such as ECS). • Expertise in Elasticsearch/OpenSearch, Splunk, or Solr. Strong proficiency in building visualization dashboards (Kibana, Grafana) and managing telemetry collectors (OpenTelemetry/OTel, Beats, Prometheus). • Experience in designing and scaling data pipelines for high-velocity telemetry. Essential expertise in Apache Kafka, Logstash, or streaming/orchestration frameworks (Airflow, Vector, Debezium). • Hands-on experience architecting scalable, containerized solutions using Kubernetes (K8s), Docker, and infrastructure-as-code (Terraform/Ansible). • Search & Indexing Engine Depth: Prior architecture experience with OpenSearch, Apache Solr, or Lucene core or Elasticsearch. • Strong proficiency in Python, Rust, or Bash for automating log parsing, testing, and deployment workflows. • A strong track record of engineering within highly regulated sectors (eg, MoD, Home Office, BAE, HMRC) with an active UK security clearance (SC/DV). We highly encourage applications if you bring any of the following equivalent expertise: • From Big Data/ETL to Telemetry Pipelines: If you have built high-throughput streaming pipelines using Apache Kafka, Airflow, NiFi, or Spark, you already know how to route and transform complex data feeds at scale. • From Enterprise Search to System Observability: If you have engineered core search, indexing, or sharding architectures using Apache Solr, Lucene, or OpenSearch, you possess the exact foundational knowledge needed to scale an enterprise Elasticsearch cluster. • From Cloud Infrastructure to Monitoring Fabrics: If you are a seasoned Cloud or Infrastructure Architect specializing in secure Kubernetes orchestration, Terraform automation, and air-gapped system design, you can easily pivot to defining secure logging architectures.