Data Engineer

Dice is the leading career destination for tech experts at every stage of their careers. Our client, GenAquarius LLC, is seeking the following. Apply via Dice today! Position: Data Engineer Location: 200, San Francisco, CA (Monday & Friday Remote, T,W & T onsite) Duration: 6+ month Note: Need only, who can directly on our payroll on contractual basis. If this opportunity interests you, please reply to this email with your updated resume and availability for a quick conversation. Looking forward to hearing from you. About the Role: The Data and Analytics team is seeking a Data Engineer to build and optimize data systems that power batch processing, real-time streaming, pipeline orchestration, data lake management, and data cataloging. You will have the opportunity to use your expertise in solving big data problems, design thinking, coding and analytical skills to develop core libraries, frameworks, and data pipelines that support data products and enable us to confidently leverage our petabyte-scale data. We re looking for talented Data Engineers passionate about building new data driven solutions with the latest Big Data technology. What you ll Do: • You will build systems, core libraries and frameworks that power our batch and streaming Data and ML applications. The services you build will integrate directly with LendingClub s products, opening the door to new features. • You will work with modern data technologies such as Hadoop, Spark, DBT, Dagster/Airflow, Atlan, Trino, etc., modern data platforms such as Databricks and Snowflake and cloud technologies across AWS stack. • Build data pipelines that transform raw data into canonical schema representing business entities and publish it into the Data Lake • Implement internal process improvements: automating manual processes, optimizing data delivery, reducing cloud costs, redesigning infrastructure for greater scalability, etc. • Work with stakeholders including the Business, Product, Program and Engineering teams to deliver required data in time with high quality at reasonable cost • Implement processes and systems to monitor Data Quality, Observability, Governance and Lineage. • Support operations to manage the production environment and help in resolving production issues with RCA • Write unit/integration tests, adopt Test-driven development, contribute to engineering wiki, and document design/implementation etc. About you: • 5+ years of experience and a bachelor's degree in computer science or a related field; or equivalent work experience • Working experience of distributed systems Hadoop, Spark, Hive, Kafka, DBT and Airflow/Dagster • At least 2 years of production coding experience in data pipeline implementation in Python • Experience working with public cloud platforms, preferably AWS • Experience working with Databricks and/or Snowflake • Experience in Git, JIRA, Jenkins, shell scripting • Familiarity with Agile methodology, test-driven development, source control management and test automation • Experience supporting and working with cross-functional teams in a dynamic environment • You have excellent collaborative problem solving and communication skills and are empathetic to others • You believe in simple and elegant solutions and give paramount importance to quality • You have a track record of building fast, reliable, and high-quality data pipelines