Data AI Engineer (Databricks)

Position: Data & AI Automation Engineer (Databricks) Location: Austin, TX (Locals preferred - Hybrid 3 Days Onsite Weekly) Duration: Long-Term Contract Required Skills: Azure Databricks, Azure Cloud, Generative AI, LLMs Note: Looking for Permanent / Visa Independent Consultant s Overvie w:We are seeking an experienced Data & AI Automation Engineer to lead the development of intelligent automation solutions within an Azure Databricks ecosystem. This role focuses on leveraging Large Language Models (LLMs), AI agents, and advanced automation techniques to enhance data engineering productivity, accelerate software delivery, and improve data quality processe s. The ideal candidate will combine strong data engineering expertise with AI-driven automation capabilities to build scalable solutions that streamline development, testing, governance, and operational activities across the data platform. Working closely with architects, engineers, and platform teams, this individual will help drive innovation through intelligent agent frameworks and automated workflo ws. Key Responsibilit • ies:Architect and develop AI-powered agents using LLMs, rule-based logic, or hybrid approaches to automate data engineering operations within Azure Databri • cks.Create automation solutions that generate, optimize, and refactor PySpark and SQL code to improve development efficiency and maintainabil • ity.Design intelligent testing and validation agents to automate data quality assessments, reconciliation processes, and QA activit • ies.Implement automated governance capabilities including metadata management, lineage tracking, compliance monitoring, and policy enforcem • ent.Develop autonomous CI/CD functions such as test creation, deployment verification, release validation, and recovery/rollback procedu • res.Build monitoring and diagnostic agents capable of detecting anomalies, identifying performance issues, and supporting root-cause investigati • ons.Establish prompt engineering standards, reusable templates, and agent orchestration patterns tailored to enterprise data platfo • rms.Partner with data engineering and architecture teams to identify automation opportunities and prioritize high-value initiati • ves.Define and enforce best practices for AI agent lifecycle management, including development, testing, deployment, observability, and maintena • nce.Produce technical documentation, workflow diagrams, operational procedures, and knowledge-transfer materi • als.Evaluate emerging AI technologies, agent frameworks, and automation tools to support continuous platform improvem ent. Required Experi • ence:10+ years of experience in Data Engineering, Software Engineering, or a related technical discip • line.Extensive hands-on experience with modern cloud data platforms such as Azure Databricks, Snowflake, or similar technolo • gies.Proven background building AI-driven automation solutions utilizing LLMs, prompt engineering methodologies, and agent orchestration framew • orks.Strong programming expertise in Python and advanced SQL development within production environm • ents.Practical experience working with Azure Databricks, including notebooks, workflows, jobs, Delta Live Tables (DLT), and related serv • ices.Experience implementing CI/CD and Continuous Testing (CT) pipelines using tools such as GitHub Actions and Azure De • vOps.Demonstrated success creating automated testing frameworks, data validation solutions, or quality assurance proce • sses.Previous experience supporting highly regulated, compliance-driven, or risk-sensitive environments is advantag eous. Technical S • kills:Advanced Python development skills with a strong foundation in software engineering principles, testing methodologies, documentation standards, and source control prac • tices.Expertise in SQL and PySpark performance tuning and optimiz • ation.Hands-on experience integrating LLM services, AI APIs, prompt engineering techniques, and agent frameworks such as Databricks Agent Framework, LangChain, AutoGen, OpenAI, or Azure O • penAI.Deep understanding of Azure Databricks, including Delta Lake, workflows, asset bundles, and enterprise-scale data proce • ssing.Experience building workflow automation solutions using orchestration platforms, serverless technologies, or event-driven architec • tures.Strong analytical skills with the ability to break down complex business and technical processes into scalable automation compo • nents.Excellent communication skills with the ability to explain technical designs, AI agent architectures, and automation strategies to both technical and non-technical stakeho lders. Edu • cation:Bachelor’s degree in Computer Science, Software Engineering, Data Science, Information Technology, or a related • field.Master’s degree pre ferred.