Senior Data Engineer (Azure/Databricks / Spark)

Job Title: Senior Data Engineer (Azure/ Databricks / Spark) Duration: 7 months Location: Austin, TX Overview: The Worker is responsible for developing, maintaining, and optimizing big data solutions using the Databricks Unified Analytics Platform. This role supports data engineering, machine learning, and analytics initiatives within this organization that relies on large-scale data processing. Duties Include • Designsing and developing scalable data pipelines • Implementing ETL/ELT workflows • Optimizing Spark jobs • Integrating with Azure Data Factory • Automating deployments • Collaborating with cross-functional teams • Ensuring data quality, governance, and security. Required Skills • 4+ years of experience Implement ETL/ELT workflows for both structured and unstructured data • 4+ years of experience Automate deployments using CI/CD tools • 4+ years of experience Collaborate with cross-functional teams including data scientists, analysts, and stakeholders • 4+ years of experience Design and maintain data models, schemas, and database structures to support analytical and operational use cases • 4+ years of experience Evaluate and implement appropriate data storage solutions, including data lakes (Azure Data Lake Storage) and data warehouses • 4+ years of experience Implement data validation and quality checks to ensure accuracy and consistency • 4+ years of experience Contribute to data governance initiatives, including metadata management, data lineage, and data cataloging • 4+ years of experience Implement data security measures, including encryption, access controls, and auditing; ensure compliance with regulations and best practices • 4+ years of Proficiency in Python and R programming languages • 4+ years of experience Strong SQL querying and data manipulation skills • 4+ years of experience Experience with Azure cloud platform • 4+ years of Experience with DevOps, CI/CD pipelines, and version control systems • 4+ years of Working in agile, multicultural environments • 4+ years of Strong troubleshooting and debugging capabilities • 3+ years of Design and develop scalable data pipelines using Apache Spark on Databricks • 3+ years of Optimize Spark jobs for performance and cost-efficiency • 3+ years of Integrate Databricks solutions with cloud services (Azure Data Factory) • 3+ years of Ensure data quality, governance, and security using Unity Catalog or Delta Lake • 3+ years of Deep understanding of Apache Spark architecture, RDDs, DataFrames, and Spark SQL • 3+ years of Hands-on experience with Databricks notebooks, clusters, jobs, and Delta Lake • 1+ years of Knowledge of ML libraries (MLflow, Scikit-learn, TensorFlow). Preferred • 1+ years of Databricks Certified Associate Developer for Apache Spark. Preferred • 1+ years of Azure Data Engineer Associate. Preferred