Data Engineer

Job Description: Senior Azure Databricks Engineer Location Options (Onsite Required): Dallas, TX, San Jose, CA, Pleasanton, CA Experience: 9+ Years About the Role We are seeking a highly experienced Senior Azure Databricks Engineer to support digital transformation initiatives for our client. The ideal candidate has strong expertise in Azure cloud services, Databricks, Spark, large-scale data engineering, and end-to-end pipeline development. This is an onsite role requiring candidates to work from any of the client locations listed above. Responsibilities • Design, build, and optimize end-to-end data pipelines using Azure Databricks and Apache Spark. • Develop scalable ETL/ELT processes for structured and unstructured data. • Integrate Databricks with various Azure services (ADLS, ADF, Azure SQL, Synapse, Key Vault, etc.). • Implement Delta Lake features—time travel, ACID transactions, schema evolution, and optimization. • Perform advanced data transformation, cleansing, aggregation, and feature engineering. • Optimize Spark jobs for performance, reliability, and cost efficiency. • Analyze and troubleshoot complex data issues in production environments. • Collaborate with data architects, analysts, and business teams to define requirements & deliver solutions. • Follow CI/CD best practices for data engineering using Git, Azure DevOps, or similar tools. • Document solutions, create technical specifications, and support code reviews. Required Skills • 9+ years of professional Data Engineering experience. • Strong hands-on expertise with Azure Databricks and Apache Spark (PySpark / Scala). • Strong experience with Azure cloud services: • Azure Data Lake (ADLS) • Azure Data Factory (ADF) • Azure SQL / Synapse • Azure Key Vault • Strong SQL development skills (complex queries, window functions, tuning). • Experience with Delta Lake features and Databricks workspace management. • Experience building and running pipelines on CI/CD frameworks (Azure DevOps preferred). • Strong understanding of data modeling, partitioning, performance tuning, and job orchestration. • Experience working with large datasets in distributed environments. • Excellent communication and problem-solving skills. Education: Bachelor’s or Master’s degree in Computer Science, Data Engineering, or related field.