Data Engineer

Databricks Engineer Plano, TX (Day 1 onsite) 12+ months contract Job Description: Required Skills & Qualifications: • Strong hands-on experience with Databricks, Apache Spark, PySpark, and Spark SQL. • Proficiency in Python, SQL, and data modeling concepts. • Experience working with Azure Data Lake Storage (ADLS), Azure Synapse, Azure Data Factory, or equivalent cloud services (AWS/GCP). • Solid understanding of ETL/ELT design patterns, data warehousing, and data lakehouse architectures. • Experience with Delta Lake, Unity Catalog, and MLflow is a plus. • Knowledge of CI/CD pipelines and infrastructure-as-code (IaC) tools such as Terraform or ARM templates. • Familiarity with Git-based version control and Agile delivery methodologies. • Excellent problem-solving, debugging, and performance tuning skills. Key Responsibilities: • Design, build, and maintain data pipelines and ETL/ELT workflows using Databricks (Spark, PySpark, SQL, Delta Lake). • Develop scalable data ingestion, cleansing, and transformation processes from various structured and unstructured data sources. • Implement and optimize Delta Lake for data versioning, reliability, and performance. • Work closely with data architects, data scientists, and BI teams to deliver high-quality, production-grade datasets. • Tune and optimize Spark clusters and jobs for performance and cost efficiency. • Integrate Databricks with Azure Data Lake Storage (ADLS), Azure Synapse, Event Hubs, and other Azure services. • Manage job scheduling, orchestration, and monitoring (e.g., using Azure Data Factory, Airflow, or Databricks Workflows). • Implement data quality checks, logging, and error handling mechanisms. • Follow DevOps and CI/CD practices for data engineering projects (e.g., Git, Azure DevOps, Jenkins). • Ensure compliance with data governance, security, and privacy standards.