Big Data Cloud Engineer

Title: Lead Big Data Cloud Engineer Location: Remote. Candidates must be willing to work onsite in Reston, VA or Washington, DC, once per month for all-hands meetings. Overview & Responsibilities: We are seeking a Lead Big Data Cloud Engineer for a crucial production support role with a leading healthcare services client. You will be responsible for the administration and support of the "Enterprise Data Hub" (EDH), a complex Big Data platform built on Cloudera (CDP) and running in the AWS cloud. This is a hands-on role for a lead engineer experienced with the Cloudera ecosystem (including Kafka, Nifi, Hbase, and Spark) who will ensure the health, performance, and security of 15 different data products and real-time data pipelines. You will… • Provide lead-level production support for the Cloudera (CDP) platform running on AWS. • Monitor the health of all Hadoop daemon services and Cloudera services using Cloudera Manager. • Administer and troubleshoot the real-time data pipeline, including Apache Kafka (brokers, topics, streams) and Apache Nifi (flow management, registry). • Support and troubleshoot other ecosystem components, including Hbase and Solr. • Write and maintain shell scripts for health checks and monitoring. • Support data migrations and perform incremental updates/upgrades to the Cloudera environment. • Manage job workflows (Oozie, Hue) and implement security policies (Ranger). • Participate in a 24/7 on-call rotation to support the production platform. Required Qualifications: • Must be available for a mandatory in-person interview in Reston, VA, or Washington, D.C. • Strong experience supporting Cloudera applications (CDP preferred) running in a Cloud (AWS) environment. • Strong administration and troubleshooting skills with Apache Kafka. • Strong administration and troubleshooting skills with Apache Nifi. • Experience supporting Hbase and Solr. • Experience writing Hive and Impala queries. • Experience with shell scripting for monitoring and automation. • Experience in a 24/7 production support role with on-call rotations. Preferred Qualifications: • Experience with Cloudera migrations (CDH to CDP). • Experience with Mongo DB. • Experience with Cloud Disaster Recovery strategies. • Experience with Flume, Kudu, or Spark. • Experience implementing security with Kerberos and Ranger.