Hadoop Engineer - ODP Platform

Role Title: Hadoop Engineer / ODP Platform Location: Birmingham / Sheffield - Hybrid working with 3 days onsite per week End Date: 28/11/2025 Role Overview We are seeking a highly skilled Hadoop Engineer to support and enhance our Operational Data Platform (ODP) deployed in an on-premises environment. The ideal candidate will have extensive experience in the Hadoop ecosystem, strong programming skills, and a solid understanding of infrastructure-level data analytics. This role focuses on building and maintaining scalable, secure, and high-performance data pipelines within enterprise-grade on-prem systems. Key Responsibilities • Design, develop, and maintain data pipelines using Hadoop technologies in an on-premises infrastructure. • Build and optimise workflows using Apache Airflow and Spark Streaming for real-time data processing. • Develop robust data engineering solutions using Python for automation and transformation. • Collaborate with infrastructure and analytics teams to support operational data use cases. • Monitor and troubleshoot data jobs, ensuring reliability and performance across the platform. • Ensure compliance with enterprise security and data governance standards. Required Skills & Experience • Minimum 5 years of experience in Hadoop and data engineering. • Strong hands-on experience with Python, Apache Airflow, and Spark Streaming. • Deep understanding of Hadoop components (HDFS, Hive, HBase, YARN) in on-prem environments. • Exposure to data analytics, preferably involving infrastructure or operational data. • Experience working with Linux systems, shell scripting, and enterprise-grade deployment tools. • Familiarity with monitoring and logging tools relevant to on-prem setups. Preferred Qualifications • Experience with enterprise ODP platforms or similar large-scale data systems. • Knowledge of configuration management tools (e.g., Ansible, Puppet) and CI/CD in on-prem environments. • Understanding of network and storage architecture in data centers. • Familiarity with data security, compliance, and audit requirements in regulated industries.