Data Engineer – ETL & Streaming - Remote - 6 month Contract Overview: My Client is seeking a highly skilled Data Engineer with strong expertise in ETL pipelines, real-time streaming, and telemetry data processing. This role involves building and optimising data pipelines, instrumenting systems for observability, and applying machine learning to streaming telemetry for actionable insights. Core Responsibilities: Programming & Statistical Computing • Expert in Python for data manipulation, statistical analysis, and ML model development. • Experience with Scala or Java (preferred for big data frameworks such as Apache Spark, Kafka, and backend systems). OpenTelemetry & Observability • In-depth understanding of OpenTelemetry architecture and OTLP (OpenTelemetry Protocol). • Experience in instrumenting and collecting telemetry from: • Virtual Machines • Middleware (e.g., IIS, Apache) • Hypervisors • Databases • Skilled in structuring and normalizing telemetry data for downstream analysis. Streaming Data & ML Integration • Proven experience with OLTP and real-time streaming data pipelines. • Ability to correlate telemetry across multiple layers (infrastructure, applications, databases). • Application of ML models to streaming data for actionable insights. Telemetry Data Modeling • Design and implement structured telemetry schemas. • Strong grasp of distributed tracing, metrics, and log correlation. ML-Driven Observability • Apply ML techniques (e.g., anomaly detection, predictive analytics) to observability datasets. • Develop feedback loops to trigger automated responses based on telemetry insights. Collaboration & Communication • Effectively communicate technical telemetry and ML insights to cross-functional teams. • Partner with DevOps, SRE, and platform teams to enhance observability maturity. Mandatory Skills • Flask • Apache Spark • Python • Nginx • Django • Jupyter Notebook • Scala • Apache Spark (Scala) • Python – Data Science Fully Remote | Up to £380/day (Outside IR35, Umbrella) If this sounds like you, please apply directly.