Azure Databricks Engineer

DataStaff, Inc is seeking an Azure Databricks Engineer for a long-term contract opportunity with one of our direct clients located in Raleigh, NC. • This position is hybrid; 1 day per week will be on-site Responsibilities: • Work with existing staff to plan and design ETL pipelines and product solutions using Azure Databricks. • Create resilient processes to ingest data from a variety of on-prem and cloud transactional databases and APIs. • Developing business requirements, facilitating change management documentation, and actively collaborating with stakeholders. • Work closely with a development technical lead and discuss all aspects of the design and planning with the development team. • Research and engineer repeatable and resilient ETL workflows using Databricks notebooks and Delta Live Tables for both batch and stream processing • Collaborate with business users to develop data products that align with business domain expectations • Work with DBAs to ingest data from cloud and on-prem transactional databases • Contribute to the development of the Data Architecture - Transportation: • By following practices for keeping sensitive data secure • By streamlining the development of data products for use by data analysts and data scientists • By developing and maintaining documentation for data engineering processes • By ensuring data quality through testing and validation • By sharing insights and experiences with stakeholders and engineers Required Skills: • 5 Years - Excellent interpersonal skills as well as written and communication skills. • 2 Years - Able to write clean, easy-to-follow Databricks notebook code • 2 Years - Deep knowledge of data engineering best practices, data warehouses, data lakes, and the Delta Lake architecture • 2 Years - Good knowledge of Spark and Databricks SQL/PySpark • 2 Years - Technical experience with Azure Databricks and cloud providers like AWS, Google Cloud, or Azure • 2 Years - In-depth knowledge of OLTP and OLAP systems, Apache Spark, and streaming products like Azure Service Bus • 2 Years - Good practical experience with Databricks Delta Live Tables Desired Skills: • 7 Years - Knowledge of object-oriented languages like C#, Java, or Python This opportunity is available on a corp-to-corp basis or as a W2 position with a competitive benefits package. DataStaff, Inc. offers medical, dental, and vision coverage options as well as paid vacation, sick, and holiday leave. As many of our opportunities are long-term, we also have a 401k program available for employees after 6 months.