Data Engineer – Enterprise Data Hub (EDH) / Cloudera

Data Engineer – Enterprise Data Hub (EDH) / Cloudera Role: Data Engineer – EDH / Cloudera Location: UK Contract: 12 month contract Clearance: SC Clearance required / SC eligible We are looking for experienced Data Engineers to join a large-scale UK Government data programme, working within an established Enterprise Data Hub (EDH) environment. This is a specialist requirement and previous hands-on Enterprise Data Hub (EDH) experience is mandatory. The successful Data Engineer will work across data ingestion and loading services, building, maintaining and supporting data feeds into a large-scale Cloudera/Hadoop data platform. Responsibilities • Build and maintain data ingestion pipelines and data feeds into an Enterprise Data Hub • Develop and support Data Loading Service (DLS) pipelines • Perform data validation, cleansing, structure and format checking • Develop ETL and transformation processes using technologies including Hive, Pig and Spark • Use ingestion technologies including Apache NiFi, Sqoop and Flume • Monitor production data feeds and troubleshoot pipeline failures • Maintain data quality, completeness and accuracy • Work with sensitive and highly governed datasets • Support secure data processing, tokenisation and entity-protection controls • Work within an Agile delivery environment Required Experience • Previous Enterprise Data Hub (EDH) experience – essential • Strong commercial Data Engineering experience • Strong Cloudera / Hadoop experience • Experience developing large-scale data ingestion and data feed pipelines • Experience with technologies including: • Hadoop / HDFS • Cloudera CDH • Hive • Pig • Sqoop • Apache NiFi • Flume • Spark / Spark Streaming • Experience with HBase and/or Cassandra • Python, Scala and/or Java experience • Strong understanding of data validation, cleansing and data quality • Experience working with sensitive or highly governed data Experience with Data Loading Services (DLS), Protegrity, tokenisation, Kerberos or Entity Protection Controls would be particularly advantageous. Security Clearance Due to the nature of the programme, candidates must either hold SC Clearance or be eligible and willing to undergo SC vetting. If you have previous Enterprise Data Hub (EDH) experience and would like to discuss the opportunity, please apply with your latest CV.