Data Engineers - Databricks - W2

Data Engineers - Databricks Location: Candidates located in the Tri-State area (NY, NJ, CT) will be given first preference, as the role could require onsite presence 1–2 days per week in Philadelphia. However, we are also open to remote candidates based in the Eastern Time Zone (EST). Anticipated Contract Duration: likely until end of year, potential extensions based on project completion. Project Details: The team of Data Engineers will play a crucial role in a large Data Engineering project beginning on the GIS Software team at a large, enterprise-level telecommunications provider. These data engineers will be focused on optimizing the current Databricks environment through building and developing data transformations, designing and implementing graph data integration patterns and relationship mapping strategies, implementing DABs for streamlined deployment workflows, and Medallion Architecture Implementation. In addition, these data engineers should have experience with Databricks Asset Bundling for CI/CD automation, conducting code reviews for adherence to development standards, and be comfortable participating in architectural decisions and technical solution design for this project. Candidates should have experience with: • Databricks Asset Bundling to support CI/CD automation • Conducting code reviews to ensure adherence to development standards • Contributing to architectural discussions and technical solution design The ideal candidate will focus on optimizing the current Databricks environment by: • Building and enhancing data transformations • Designing and implementing graph data integration patterns and relationship mapping strategies • Supporting the implementation of Medallion Architecture Data Engineer Qualifications: • 5+ years of experience in data engineering with at least 3 years focused on the Databricks platform • Hands-on experience with Databricks Asset Bundles (DABs) for CI/CD automation • Proven experience implementing Medallion architecture patterns in production environments • Strong proficiency in Databricks platform capabilities and data processing frameworks • Experience with data lake management concepts and transactional data operations Nice to Have Skills: • Databricks certified professional credentials (Data Engineer Associate/Professional) • Experience with real-time streaming data processing using Databricks Structured Streaming • Knowledge of machine learning workflows and MLOps practices within Databricks • Experience with Infrastructure as Code tools and deployment automation • Familiarity with containerization technologies and modern DevOps practices • Previous consulting or managed services experience • Experience with data visualization tools and business intelligence platforms • Knowledge of graph algorithms and network analysis techniques Data Engineer responsibilities: • Design and implement robust data pipelines using Databricks platform capabilities • Develop and maintain ETL/ELT processes following best practices for performance and reliability • Build and optimize data transformations within Databricks environments • Design and implement graph data integration patterns and relationship mapping strategies • Implement and manage Databricks Asset Bundles (DABs) for streamlined deployment workflows • Configure and maintain CI/CD pipelines using Asset Bundles for development, staging, and production environments • Participate in architectural decisions and technical solution design • Stay current with Databricks platform updates and emerging data engineering technologies