Databrick Engineer

Job Title: Senior Databricks Engineer (Python/PySpark) Location: New York, NY (Hybrid – 3 days onsite, 2 days remote) Experience Level: 10+ years Employment Type: Contract Rate: $63/hr Department: Data Engineering / Analytics About the Role We are seeking a highly skilled Senior Databricks Engineer with deep expertise in Python, PySpark, and cloud-based data engineering. The ideal candidate will design, build, and optimize large-scale data pipelines and analytics solutions using Databricks on a modern data lakehouse architecture. This role requires a hands-on technical leader who can work cross-functionally with data architects, analysts, and business stakeholders to deliver high-quality data solutions in a hybrid working environment. Key Responsibilities • Design, build, and maintain ETL/ELT data pipelines using Databricks, Python, and PySpark. • Develop and manage data lakehouse architecture using Delta Lake. • Integrate structured and unstructured data from multiple on-prem and cloud sources. • Optimize Spark jobs for performance, scalability, and cost efficiency. • Collaborate with data scientists and analysts to support advanced analytics and machine learning workloads. • Implement data quality, lineage, and governance frameworks within Databricks. • Automate workflows and orchestration using Databricks Workflows, Airflow, or Azure Data Factory. • Manage CI/CD pipelines for data projects using Git-based workflows. • Ensure data security, access controls, and compliance with organizational standards. • Mentor junior engineers and contribute to best practices in data engineering. Required Skills & Qualifications • Bachelor’s or Master’s degree in Computer Science, Information Systems, or a related field. • 10+ years of overall experience in data engineering with 3+ years hands-on in Databricks. • Strong proficiency in Python and PySpark for large-scale data transformations. • In-depth understanding of Spark architecture, optimization, and cluster management. • Experience building and managing Delta Lake solutions and data lakehouse architectures. • Expertise with SQL for data manipulation and performance tuning. • Strong knowledge of cloud platforms — preferably Azure Databricks (AWS/GCP also acceptable). • Familiarity with data orchestration tools such as Airflow, ADF, or Databricks Jobs. • Experience with CI/CD pipelines and tools (Azure DevOps, GitHub Actions, Jenkins, etc.). • Hands-on experience implementing data governance, cataloging, and security (e.g., Unity Catalog). Preferred Skills • Exposure to streaming data (Kafka, Event Hubs, Kinesis). • Experience with MLOps and integrating machine learning workflows in Databricks. • Knowledge of data warehousing concepts and BI tools (Power BI, Tableau). • Certifications such as Databricks Certified Data Engineer Professional or Azure Data Engineer Associate. Soft Skills • Strong analytical and problem-solving mindset. • Excellent communication and stakeholder management skills. • Ability to lead data engineering initiatives end-to-end. • Team player with mentorship and knowledge-sharing capabilities.