Job Title:Azure Databricks Lead Location:Erlanger, Kentucky (Onsite) Contract Overview: We are seeking a highly skilled and motivated Senior Technical Lead to drive the development and deployment of Azure Databricks-based solutions. This role requires deep expertise in: • Python programming • Databricks administration • CI/CD pipelines for Databricks artifacts • Delta Live Tables • Auto Loader • Unity Catalog • Databricks Asset Bundles The ideal candidate will bring a strong technical background, proven team leadership experience, and a passion for building high-quality, scalable solutions. Key Responsibilities: Python Programming • Lead the development of robust, scalable, and efficient Python code for data processing, transformation, and analysis within Databricks. • Ensure adherence to coding standards, quality, and performance best practices. Azure Databricks Administration • Oversee platform configuration, resource management, cluster optimization, and monitoring. • Implement best practices for managing workspaces, libraries, and notebooks. CI/CD for Databricks Artifacts • Design and implement CI/CD pipelines to automate deployment of Databricks artifacts (notebooks, libraries, jobs, Delta tables). • Utilize tools like Azure DevOps, GitHub Actions, or Jenkins. Delta Live Tables • Design and manage Delta Live Tables pipelines for real-time data processing. • Ensure data quality, reliability, and optimal performance. Auto Loader • Implement and optimize Auto Loader for scalable, fault-tolerant data ingestion from cloud storage. Unity Catalog • Configure and manage Unity Catalog for centralized data governance, access control, lineage, and compliance. Databricks Asset Bundles • Leverage Databricks Asset Bundles to manage and share reusable components across teams and projects. Team Leadership • Mentor and guide junior team members. • Foster a collaborative environment that encourages innovation and personal growth. • Provide technical leadership and support to drive successful outcomes. Collaboration & Communication • Work closely with data engineers, data scientists, and business analysts to understand requirements. • Deliver end-to-end solutions. • Communicate effectively with technical and non-technical stakeholders. Required Qualifications: Python Programming • 5+ years of experience focused on data engineering and data science workflows. • Expertise in designing and building scalable ETL pipelines with libraries such as Pandas, NumPy, PySpark, requests. • Proficiency in debugging, optimizing, and maintaining Python code in distributed environments. Azure Databricks • 3+ years of hands-on experience in production Databricks environments. • Strong knowledge of clusters, workspaces, and runtime configurations. • Ability to optimize and troubleshoot Spark-based jobs and notebooks. CI/CD Pipelines • Proven experience setting up and managing CI/CD pipelines for Databricks artifacts. • Familiarity with Git and tools such as Azure DevOps, GitHub Actions, or Jenkins. • Experience integrating Databricks with external version control repositories. Delta Live Tables • Strong experience designing, managing, and optimizing DLT pipelines for real-time, reliable data processing. • Knowledge of automated data quality checks and validation within DLT. Auto Loader • Practical experience with Auto Loader for real-time ingestion from cloud storage (Azure Blob Storage, ADLS Gen2). • Understanding of performance optimization best practices. Unity Catalog • In-depth experience implementing Unity Catalog for centralized data governance. • Proven ability to set up fine-grained access controls and manage permissions.