Big Data Engineer

Hybrid 3 days a week in either locations: Tysons Corner, VA or Rockville, MA! About the Position / Current Initiatives: The position involves designing, developing, and optimizing large-scale data processing systems. The role requires working on petabyte-scale data with Big Data technologies, troubleshooting resource limitations, and leveraging AI tools for prompt engineering and workflow improvements. Job Must Haves: • Experience with Big Data technologies such as Hadoop, Spark, Hive, and Trino • Strong SQL skills, including complex queries, window functions, and multi-table joins • Cloud experience, specifically with AWS services like S3, Glue, and EMR • Proficiency in AI tools (e.g., GitHub Copilot, ChatGPT) and prompt engineering • Hands-on experience with Apache Spark development, internals, and tuning • Programming expertise in Python or Scala • Agile methodology experience, CI/CD practices, and financial data experience preferred Job Nice to Haves: • AWS certifications • Experience managing production data pipelines/ETL systems • Experience writing test cases • Knowledge of serverless technologies and EKS