Databricks Engineer (Texas Residents Only)

Job Title: Databricks Engineer Location: Remote (Texas Residents only) Job Type: Contract Note: The person has to come visit the Austin office once a Month. Job Summary: We are seeking a skilled Databricks Engineer to join our client’s data engineering team. The ideal candidate will have strong expertise in Databricks, Apache Spark, and cloud data platforms. You will be responsible for building and optimizing scalable data pipelines, implementing best practices for data processing, and enabling advanced analytics through efficient data infrastructure on the cloud. Key Responsibilities: • Design, develop, and maintain scalable ETL pipelines using Apache Spark on Databricks. • Implement and manage data lakes, Delta Lake, and data warehouse environments on cloud platforms (Azure, AWS, or GCP). • Optimize Databricks cluster configurations and performance to ensure cost efficiency and scalability. • Collaborate with data scientists, analysts, and business stakeholders to translate data requirements into reliable technical solutions. • Develop and maintain CI/CD pipelines for data workflows using Azure DevOps, GitHub Actions, or similar tools. • Integrate structured and unstructured data from multiple on-prem and cloud sources. • Apply data quality, testing, and security best practices throughout the data lifecycle. • Contribute to architecture discussions and documentation of technical workflows and processes. • Implement automation and infrastructure-as-code (IaC) solutions using tools like Terraform or ARM templates. Required Skills and Qualifications: • Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field. • 8+ years of experience in data engineering or big data development. • Strong hands-on experience with Databricks, Apache Spark, and Delta Lake. • Experience with at least one major cloud platform (Azure, AWS, or GCP). • Proficiency in Python and SQL (Scala is a plus). • Solid understanding of data modeling, data warehousing, and distributed data processing. • Experience with CI/CD, DevOps, and infrastructure automation tools (Terraform, ARM templates, etc.). • Strong analytical, problem-solving, and communication skills.