Databricks Lead

About Aspire: Aspire Systems is a $170 million global technology services firm serving as a trusted technology partner for our customers. We work with some of the world's most innovative enterprises and independent software vendors, helping them leverage technology and outsourcing in our specific areas of expertise. It is a responsible and diverse organization of over 4,600 people across the globe. Our core philosophy of "Attention. Always." communicates our belief in lavishing care and attention on our customers and employees. Title: Databricks Lead Location: St. Louis, MO / Non Local who are willing to relocation is fine Job Type: 6-8 months Contract to hire Experience: 10 – 12+ years Rate: $85 per hour on C2H Job Description: We are looking for a skilled and experienced Databricks Lead to drive data engineering initiatives, lead solution architecture, and support large-scale data pipelines using Apache Spark on Databricks, Kafka, Stelo, and Azure Data Factory (ADF). The ideal candidate will possess a strong foundation in cloud data platforms, streaming technologies, and a leadership mindset to mentor and guide teams in delivering high-quality solutions. Key Responsibilities: • Lead the design, development, and deployment of scalable data pipelines using Azure Databricks. Integrate real-time and batch data sources using Apache Kafka and ADF. • Implement data transformations, enrichment, and quality checks using PySpark/Scala within the Databricks environment. • Design and develop integrations with Stelo for structured data loading and transformation (if applicable). • Collaborate with data scientists, analysts, and other engineering teams to deliver business-critical insights. • Optimize pipeline performance, cost, and scalability in the Azure cloud environment. • Define best practices for data ingestion, processing, storage, and governance. • Lead and mentor a team of data engineers, conducting code reviews and ensuring development standards. • Support troubleshooting and incident management for data-related issues in production. Required Skills and Qualifications: • 8+ years of experience in data engineering with at least 2 years in a lead role. • Strong hands-on experience with Azure Databricks, Apache Spark, and Delta Lake. • Proven experience integrating and managing streaming data using Apache Kafka. • Experience using Azure Data Factory for orchestrating and scheduling ETL/ELT processes. • Familiarity or experience with Stelo for metadata-driven data movement (or similar tools). • Proficiency in Python, SQL, and optionally Scala. • Good understanding of data warehousing concepts, data lakes, and big data architecture. • Knowledge of Azure ecosystem including ADLS, Synapse, Key Vault, etc. • Excellent analytical, problem-solving. Preferred Qualifications: • Azure certifications such as Azure Data Engineer Associate or Azure Solutions Architect. • Experience with DevOps, CI/CD pipelines, and Infrastructure as Code (e.g., Terraform, ARM templates). • Exposure to data governance tools and practices (Purview, Unity Catalog). Disclaimer: The salary, other compensation, and benefits information is accurate as of the date of this posting. Aspire reserves the right to modify this information at any time, subject to applicable law. Our commitment to diversity and inclusion: At Aspire Systems, we foster a work culture that appreciates diversity and inclusiveness. We understand that our multigenerational workforce belongs to different regions, cultures, economic backgrounds, race, gender, ethnicity, education, personality, and religion.