Senior Azure DataBricks Engineer Lead

Job Title: Senior Azure Databricks Engineer Lead Location: Remote Job Type: Full Time. Job Description: The Azure Databricks Platform Architect/Admin will be responsible for designing, implementing, and managing our Azure Databricks platform. The ideal candidate will have a strong background in Azure Databricks, data engineering, and data architecture, with a focus on building scalable and secure data platforms. The successful candidate will work closely with our data engineering and data science teams to ensure seamless integration of data pipelines and Lakehouse architecture. This role should ensure coding standard and coding migration to higher environment and CI/CD Design, implement, and manage our Azure Databricks platform, ensuring high availability, scalability, and security. Configure and optimize Databricks clusters, jobs, and notebooks for performance and cost-effectiveness. Maintain up to date architecture environment. Works with Azure and Databricks to resolved platform issues/tickets. Optimize cost / performance Collaborate with data engineering teams to design and implement/migrate data pipelines using using industry best practice. Automate manual admin tasks. Develop and maintain data Lakehouse architecture, ensuring data quality, integrity, and security. Work with Azure IAM and Engineering team to come up with RBAC for Databricks tables. Design and manage Lakehouse and sandbox environments for data science teams, ensuring data governance and security. Collaborate with data science teams to integrate data pipelines and Lakehouse architecture. Ensure Azure Databricks platform security and compliance with organizational security policies and regulatory requirements (e.g., HIPAA,NIST,GDPR). Implement data encryption, access controls, and auditing. Develop and maintain technical documentation for Azure Databricks platform, data pipelines, and Lakehouse architecture. Share knowledge and best practices with data engineering and data science teams.