Splunk Developer

Splunk Developer – Technical Lead (ITSI) Location: Edinburgh (Hybrid – 2 days onsite per week) Contract Role Job Title: Splunk Cloud & IT Service Intelligence (ITSI) Engineer – Advanced Engineering Focus We are seeking a highly skilled Splunk Cloud & ITSI Engineer with strong software engineering and coding capability to design, build, and maintain enterprise-grade observability, analytics, and service health platforms. This is a hands-on engineering role, requiring strong depth in Splunk SPL development, automation, data engineering, and scripting/programming to build scalable monitoring solutions across a complex enterprise environment. The role combines ITSI service modelling, advanced SPL development, automation engineering, and Splunk Cloud platform administration, with a strong emphasis on engineering-quality solutions rather than configuration-only work. Key Responsibilities 1. ITSI Service Engineering & Development • Design and implement ITSI service models including service trees, dependencies, KPIs, and health scoring frameworks. • Develop advanced KPI logic using complex SPL, scripted inputs, and custom calculations. • Build adaptive thresholds, SLO-based indicators, and golden signal-based monitoring. • Engineer correlation logic to detect service degradation and performance anomalies. 1. Advanced SPL Development & Engineering • Write complex, production-grade SPL queries for dashboards, alerting, correlation searches, and analytics. • Optimise SPL performance through query refactoring, acceleration, and search-time tuning. • Develop reusable SPL macros, modular search components, and reusable knowledge objects. • Debug and enhance large-scale distributed search workloads. 1. Automation, Scripting & Integration Engineering • Develop automation scripts (Python or similar) to support data ingestion, enrichment, and ITSI workflows. • Build integrations between Splunk and external systems using APIs, webhooks, and automation frameworks. • Engineer data pipelines and transformation logic for observability datasets. • Support event-driven automation and remediation workflows. 1. Splunk Cloud Engineering & Platform Support • Support Splunk Cloud architecture including ingestion pipelines, HEC, forwarders, and deployment servers. • Manage CIM alignment, data normalization, and structured onboarding of complex data sources. • Implement RBAC models, index strategies, and data lifecycle management. • Optimise platform performance, search concurrency, and workload management. 1. Observability & Analytics Engineering • Build real-time dashboards using SPL, data models, and accelerated datasets. • Develop predictive monitoring using MLTK and anomaly detection techniques. • Integrate logs, metrics, and events into unified observability views. • Implement alerting frameworks with intelligent suppression, routing, and enrichment. 1. ITSI Operations & Service Health Intelligence • Configure NEAP policies to reduce noise and improve signal quality. • Build Glass Tables, Service Analyzer views, and executive dashboards. • Design service degradation detection models and incident correlation logic. • Integrate ITSI outputs with ITSM and CMDB systems. Required Skills & Experience • 4–8+ years of hands-on experience in Splunk Enterprise / Splunk Cloud engineering environments • Strong software engineering mindset with hands-on coding/scripting ability (Python or equivalent preferred) • Expert-level proficiency in SPL (including complex, multi-stage queries and optimisation) • Strong experience designing and building ITSI service models and KPIs • Experience with automation, APIs, and scripting for integration and data processing • Deep understanding of observability principles (logs, metrics, traces, golden signals) • Experience with Splunk Cloud architecture including ingestion, indexing, RBAC, and performance tuning • Strong troubleshooting skills across search performance, ingestion pipelines, and distributed systems • Experience with MLTK, anomaly detection, or predictive analytics in observability contexts Preferred Skills • Python or similar scripting language for automation and integrations • Experience with CI/CD pipelines for observability or monitoring platforms • Splunk certifications (Admin, ITSI Admin, Architect preferred) • Experience working in large-scale enterprise or regulated environments