Big Data Developer

We’re seeking a Sr Big Data Engineer with strong hands-on expertise across the Big Data ecosystem, particularly deep experience in Hadoop, Hive, Spark, Kafka, HBase, and distributed data platforms. This is not a traditional Hadoop Administrator role focused on cluster support, nor a pure application developer position. Historically, they have seen profiles from Hadoop Admin backgrounds that lacked the engineering depth required for this opportunity. The ideal candidate should have proven experience designing, building, and optimizing near real-time and batch data pipelines, working with large-scale distributed systems, and handling complex data ingestion from sources such as message queues, REST APIs, relational databases, file shares, and multiple formats including JSON, CSV, and XML. Strong SQL capabilities across technologies such as Hive, Impala, Spark SQL, or MySQL are essential. Hands-on development expertise in Scala, Python, Shell Scripting, or similar languages is highly preferred, along with experience in Spark Structured Streaming, performance tuning of Spark/MapReduce/SQL workloads, and working within enterprise Linux environments. Exposure to tools such as Sqoop, Solr, Elasticsearch, Kibana, Jenkins, Git/Bitbucket, and Agile delivery practices will be valuable. Required Skills: • Strong SQL Skills – one or more of MySQL, HIVE, Impala, SPARK SQL • Data ingestion experience from message queue, file share, REST API, relational database, etc. and experience with data formats like json, csv, xml • Experience working with SPARK Structured steaming • Experience working with Hadoop/Big Data and Distributed Systems • Working experience with Spark, Sqoop, Kafka, MapReduce, NoSQL Database like HBase, SOLR, CDP or HDP, Cloudera or Hortonworks, Elastic Search, Kibana, etc. • Hands on programming experience in at least one of Scala, Python, PHP, or Shell Scripting, to name a few • Performance tuning experience with spark /MapReduce or SQL jobs • Experience and proficiency with Linux operating system is a must • Experience in end-to-end design and build process of Near-Real Time and Batch Data Pipelines • Experience working in Agile development process and deep understanding of various phases of the Software Development Life Cycle • Experience using Source Code and Version Control systems like SVN, Git, Bit Bucket etc. • Experience working with Jenkins and Jar management • Self-starter who works with minimal supervision and the ability to work in a team of diverse skill sets • Ability to comprehend customer requests and provide the correct solution • Strong analytical mind to help take on complicated problems • Desire to resolve issues and dive into potential issues • Ability to adapt and continue to learn new technologies is important.