Lead Databricks Data Engineer

Job Description: Lead Data Engineer (GenAI & Data Platforms) Location: Chicago, Illinois Experience: 13 + Years Employment Type: C2C Overview We are seeking an experienced Lead Data Engineer with strong expertise in AWS, Databricks, PySpark, ETL/ELT, Delta Live Tables (DLT), Generative AI, and Agentic AI. The ideal candidate will combine deep technical knowledge with strong client-facing skills, leading architecture discussions, proposing innovative solutions, developing proof of concepts (POCs), and driving successful delivery of enterprise-scale data and AI platforms. This role requires a hands-on technical leader who can understand business challenges, translate requirements into scalable solutions, mentor engineering teams, and build client confidence through effective communication and solutioning. Key Responsibilities Data Engineering Leadership • Lead the design, development, and implementation of scalable data engineering solutions on AWS and Databricks. • Architect and build modern data lakehouse platforms using industry best practices. • Design, develop, and optimize ETL/ELT pipelines using PySpark and Delta Live Tables (DLT). • Implement scalable, secure, and high-performance data architectures. • Drive data modeling, orchestration, pipeline optimization, and data quality initiatives. • Ensure reliability, scalability, security, and governance across data platforms. AI & GenAI Solutions • Lead the design and development of Generative AI and Agentic AI solutions aligned with business objectives. • Architect AI-powered applications leveraging LLMs, AI Agents, Retrieval-Augmented Generation (RAG), and orchestration frameworks. • Build and integrate AI solutions within enterprise data ecosystems. • Develop proof-of-concepts (POCs) and accelerators demonstrating business value. • Evaluate emerging AI technologies and recommend adoption strategies. Client Engagement & Solution Architecture • Understand client environments, business processes, and pain points. • Translate business requirements into scalable technical solutions. • Lead architecture workshops, solution discussions, and technical strategy sessions. • Present solution recommendations to business and technical stakeholders. • Drive technical conversations and establish trusted advisor relationships with clients. • Articulate complex technical concepts to both technical and non-technical audiences. • Build client confidence through effective communication, innovation, and successful delivery. Team Leadership • Provide technical leadership, mentoring, and guidance to engineering teams. • Review solution designs, code quality, and implementation approaches. • Drive Agile delivery practices and ensure timely project execution. • Collaborate with cross-functional teams including data engineers, architects, AI/ML engineers, product owners, and business stakeholders. Required Skills & Qualifications Experience • 12 years of experience in Data Engineering, Data Architecture, or related fields. • Proven experience leading enterprise-scale data and cloud transformation initiatives. • Experience working directly with clients and managing stakeholder expectations. Technical Skills • Strong hands-on expertise in AWS cloud services. • Extensive experience with Databricks and modern Lakehouse architectures. • Advanced proficiency in PySpark and Spark ecosystem. • Strong experience building ETL/ELT pipelines. • Expertise in Delta Live Tables (DLT). • Experience with data modeling, data warehousing, and pipeline optimization. • Knowledge of workflow orchestration tools and CI/CD practices. • Understanding of security, governance, and compliance in cloud environments. AI & Machine Learning • Strong experience in Generative AI and Large Language Models (LLMs). • Experience designing and implementing Agentic AI solutions. • Knowledge of AI orchestration frameworks and agent architectures. • Experience with AWS GenAI services and AI platform capabilities. • Familiarity with RAG architectures, vector databases, embeddings, and prompt engineering. Leadership & Communication • Strong client-facing communication and presentation skills. • Ability to lead solution discussions and architecture reviews. • Excellent stakeholder management and consulting skills. • Experience mentoring teams and driving technical excellence. • Strong problem-solving and analytical thinking capabilities. Preferred Qualifications • Experience with AWS AI/ML services and GenAI offerings. • Exposure to multi-cloud environments. • Knowledge of MLOps and AI platform engineering. • Experience implementing enterprise AI governance frameworks. • Relevant AWS, Databricks, or AI/ML certifications. Success Factors The ideal candidate will: • Understand client challenges and proactively propose innovative solutions. • Build compelling POCs to demonstrate business value. • Lead architecture and strategy discussions confidently. • Drive adoption of modern Data & AI platforms. • Act as a trusted advisor to clients and internal stakeholders. • Deliver scalable, secure, and future-ready Data and AI solutions.