Databricks Data Engineer

Dice is the leading career destination for tech experts at every stage of their careers. Our client, Purple Drive Technologies LLC, is seeking the following. Apply via Dice today! Position Summary We are seeking an experienced Data Engineer specializing in Databricks platform for a 6-month contract position based in Seattle, WA. This role focuses on designing, building, and deploying robust data pipelines and ETL processes within a cloud data platform environment. The ideal candidate will have strong expertise in Databricks, data modeling, and cross-functional collaboration to support enterprise data governance initiatives. Responsibilities DATA PIPELINE DEVELOPMENT • Design, build, and deploy data extraction, transformation, and loading (ETL) processes and pipelines • Extract data from various sources including databases, APIs, and data files • Develop and maintain scalable data pipelines within Databricks Cloud Data Platform • Ensure data quality, reliability, and performance across all pipeline processes • Implement data validation and monitoring mechanisms DATABRICKS PLATFORM MANAGEMENT • Build comprehensive data models that reflect domain expertise and meet current business needs • Ensure data models remain flexible and adaptable as business strategy evolves • Monitor and optimize Databricks cluster performance for cost-effective scaling and resource utilization • Implement and maintain Delta Lake for optimized data storage, ensuring data reliability, performance, and versioning • Leverage Databricks Unity Catalog for data governance and collaboration DEVOPS & AUTOMATION • Automate CI/CD pipelines for data workflows using Azure DevOps • Implement version control and deployment strategies for data pipelines • Ensure automated testing and quality assurance for data processes • Maintain infrastructure as code practices for data platform components DATA ARCHITECTURE & STORAGE • Demonstrate expertise in database storage concepts including data lakes, relational databases, NoSQL, Graph databases, and data warehousing • Design and implement efficient data storage solutions • Optimize data access patterns and query performance • Ensure proper data partitioning and indexing strategies COLLABORATION & COMMUNICATION • Collaborate with cross-functional teams to support data governance initiatives • Communicate technical concepts effectively to both technical and non-technical audiences • Work with business stakeholders to understand data requirements • Provide documentation and knowledge transfer for data solutions Qualifications REQUIRED EXPERIENCE • 5+ years of experience in data engineering and ETL development • 3+ years of hands-on experience with Databricks platform • Strong experience with cloud data platforms (Azure, AWS, or Google Cloud Platform) • Proven track record in building and maintaining data pipelines at scale Technical Skills • Databricks Expertise: • Databricks workspace and cluster management • Delta Lake implementation and optimization • Databricks Unity Catalog • Spark and PySpark programming • Programming & Development: • Strong coding skills in Python, Scala, or SQL • Experience with data transformation and data quality frameworks • Knowledge of data integration patterns and best practices • Database & Storage: • Data lake architecture and design • Relational databases (SQL Server, PostgreSQL, etc.) • NoSQL databases (MongoDB, Cassandra, etc.) • Graph databases and data warehousing concepts • DevOps & Automation: • Azure DevOps or similar CI/CD tools • Infrastructure as Code (Terraform, ARM templates) • Git version control and branching strategies Soft Skills • Strong analytical and problem-solving abilities • Excellent written and verbal communication skills • Ability to explain complex technical concepts to diverse audiences • Collaborative mindset for cross-functional team environments • Detail-oriented with focus on data quality and reliability Preferred Qualifications • Databricks certification (Data Engineer Associate/Professional) • Experience with real-time data processing and streaming • Knowledge of data governance frameworks and best practices • Experience with data visualization tools (Power BI, Tableau) • Background in Agile/Scrum development methodologies