Databricks Developer

Note to Applicants: We are actively reviewing all submissions! Even if you see a high number of applicants, we encourage you to apply if your skills align with the role. We value every application and are committed to finding the right fit. Job Overview: We are seeking a highly proficient Databricks Developer to design, develop and manage scalable data pipelines and advanced analytics solutions. This role is instrumental in transforming complex, raw data into actionable business intelligence through the Databricks platform. The successful candidate will leverage their expertise in big data processing and cloud environments to optimize workflows, ensure data integrity, and drive high-performance computational efficiency across the organization’s data ecosystem. Job Details • Job Title: Senior Databricks Developer • Job Location: Pleasanton, CA • Job Duration: Approximately 24 Months (June 1, 2026 – May 10, 2028) • Job Type: Contract • Industry Focus: Healthcare Core Competencies To be considered for this position, applicants must demonstrate mastery in the following: • Databricks Ecosystem: Extensive experience designing and deploying notebooks and jobs within a Databricks environment. • Apache Spark: Deep technical knowledge of Spark for large-scale data processing and ETL/ELT optimization. • Data Lakehouse Architecture: Practical experience with Delta Lake for ACID transactions and scalable metadata handling. • Technical Stack: Proficiency in SQL, Unix shell scripting, and Informatica. • Identity & Security: Functional understanding of ADFS (Active Directory Federation Services) for secure data access. Key Responsibilities • Pipeline Architecture: Design, develop, and maintain automated end-to-end data pipelines using Databricks and Apache Spark. • Process Optimization: Refactor existing ETL/ELT workflows to improve data throughput and reduce cloud consumption costs. • Integration Management: Seamlessly integrate Databricks with cloud storage solutions and enterprise data lakes to ensure a unified data source. • Data Governance: Implement and enforce rigorous data quality standards, security protocols, and governance best practices in alignment with organizational policy. • Stakeholder Collaboration: Partner with cross-functional teams, including data scientists and business analysts, to translate complex requirements into technical specifications. • Production Support: Monitor and troubleshoot production workflows, performing root cause analysis on job failures to ensure maximum uptime. Technical Depth & Standards The candidate must adhere to the following technical and regulatory frameworks: • Security Protocols: Ensure all data movement and storage comply with internal security benchmarks and ADFS authentication standards. • Geospatial Analysis: Apply advanced analytical methods to geospatial datasets to support location-based business intelligence. • Performance Tuning: Utilize Spark UI and Databricks monitoring tools to identify and resolve bottlenecks in partitioned datasets. • Compliance: Maintain documentation for data lineage and transformation logic to ensure audit readiness and operational transparency. Qualifications • Experience: Minimum of 6 to 10+ years of professional experience in data engineering or big data analytics, Apache Spark, Databricks . • Education: Relevant certifications in Databricks or Cloud Platforms (Azure/AWS/GCP) are considered a strong asset.