AWS Data Engineer with Data Bricks and DBT

Role: AWS Data Engineer with Databricks and DBT Location: Lebanon, NJ Duration: Long-Term Contract Responsibilities • Design and Development of Data Pipelines Design, build, and optimize robust ETL/ELT pipelines using AWS services (S3, Glue, Lambda) and the Databricks platform (Spark, Delta Lake, DLT). • Ingest and process large volumes of structured and semi-structured data from various sources (APIs, databases, streaming platforms like Kafka or Kinesis) into a centralized data lake or lake house. • Data Transformation and Modeling Develop and maintain data models (e.g., star/snowflake schemas, medallion architecture) optimized for analytics and BI tools using dbt (Data Build Tool). • Write complex and efficient SQL queries and Python/PySpark code for data manipulation, transformation, and validation within the Databricks environment. • Implement data quality checks, tests, and documentation as part of the dbt workflow, enforcing data governance and security standards. • Orchestration and Automation Orchestrate and monitor data workflows using Databricks Jobs or external tools like AWS MWAA (Managed Workflows for Apache Airflow). • Implement CI/CD pipelines and version control (Git) for all data engineering artifacts (code, configurations, dbt models) to ensure reliable and consistent deployments. • Performance Optimization and Operations Monitor, troubleshoot, and resolve issues in production data pipelines and environments to ensure high performance, reliability, and cost-efficiency. • Tune Spark jobs and optimize Delta Lake features (Z-Order, partitioning) to handle growing data volumes and complexity. • Collaboration and Support Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver actionable insights. • Provide expertise and guidance on data best practices, promoting a culture of data quality and governance. Must Have skills • SQLDBT core and DBT Cloud AWS (redshift) • Data bricks with AWS SQL server DB Stone branch scheduling tool • Should understand CICD, GIT • Work in an Agile environment with JIRA Other Skills required • Good to have Tableau experience • Harness devops Proficient in Linux / Unix environments