Databricks Engineer

Job Title : Databricks Engineer – Data Operations & Production Support Job Location : Seattle, WA (Hybrid) Job Duration :Long-term Contract(12+ months) Pay rate : $55-60/hr on W2 & $65-70/hr on C2C Job Description: We are seeking a highly skilled Databricks Engineer to support and maintain enterprise data platforms and large-scale data processing environments. This role is responsible for ensuring the reliability, performance, and availability of data pipelines built on Databricks, Apache Airflow, and PySpark. The ideal candidate will possess strong production support experience and the ability to troubleshoot and optimize distributed data workloads. Key Responsibilities Databricks & Spark Operations • Support and administer Databricks-based data platforms. • Monitor and maintain Spark and PySpark processing jobs. • Identify and resolve performance bottlenecks and processing failures. • Optimize workloads for improved scalability and resource utilization. Workflow Management • Manage and monitor Apache Airflow workflows and DAG executions. • Ensure successful and timely completion of scheduled data pipelines. • Troubleshoot workflow failures and implement corrective actions. Production Support • Provide L2/L3 support for enterprise data processing applications. • Perform root cause analysis for production incidents. • Implement permanent solutions for recurring issues. • Participate in on-call support and incident management activities. Data Quality & Reliability • Validate and reconcile datasets to ensure accuracy and consistency. • Support data quality controls and monitoring frameworks. • Maintain high availability of business-critical data pipelines. Unix/Linux Administration • Perform log analysis and troubleshooting activities. • Manage file systems and batch execution processes. • Utilize shell scripting and Unix commands to support operations. Collaboration & Documentation • Partner with Data Engineering, QA, and business teams. • Support enhancements and continuous improvement initiatives. • Maintain operational documentation, runbooks, and knowledge repositories. Mandatory Skills Candidates must have: • 4+ years of experience with Databricks and Spark/PySpark • Strong hands-on experience with Apache Airflow (DAG development, scheduling, monitoring, and troubleshooting) • Proficiency in Python and SQL • Experience providing L2/L3 production support for data platforms • Strong understanding of ETL/ELT frameworks and data warehousing concepts • Experience troubleshooting large-scale distributed processing environments • Knowledge of Spark performance tuning and optimization techniques • Hands-on experience with Unix/Linux commands, shell scripting, and log analysis • Experience with incident management and root cause analysis • Strong analytical and problem-solving skills Preferred Skills • Experience with Azure Databricks or AWS Databricks • Knowledge of Azure Data Services or Microsoft Fabric • Exposure to data governance and data quality frameworks • Experience with monitoring and observability tools • Understanding of CI/CD and DevOps practices • Familiarity with cloud platforms such as Azure or AWS Ideal Background • This position is best suited for candidates with a strong support and operations mindset who have extensive experience managing Databricks, Airflow, and PySpark environments in production and ensuring the stability of enterprise data pipelines. Regards, Vishwajeet Verma