Senior Databricks Architect / Lead Databricks Engineer

Senior Databricks Architect / Lead Databricks Engineer Remote (UK) £700/day Outside IR35 Initial 12-Month Contract | Long-Term Programme | Poosible Extension We're seeking an exceptional Senior Databricks Architect with a strong hands-on Data Engineering background to join a major long-term transformation programme. This is not a pure architecture role. You'll be a Databricks specialist who remains close to the technology, designing and building enterprise-scale data solutions while driving engineering excellence across critical data platforms. You'll take ownership of delivering robust, scalable, and production-grade data solutions within a modern Databricks ecosystem, ensuring platforms are governed, secure, maintainable, and optimised for long-term growth. 🔹 Key Responsibilities • Architect, build and optimise enterprise-grade data solutions within Databricks • Design and implement scalable Delta Lake data architectures following Medallion principles • Lead data engineering delivery across multiple strategic workstreams • Develop highly resilient, production-ready pipelines with strong operational controls • Drive data governance, security, lineage and compliance best practices • Collaborate with product, engineering and business teams to deliver trusted data assets • Mentor internal engineering teams and establish engineering standards 🔹 Core Workstreams 📊 Product Data Ingestion • Design and maintain large-scale ingestion pipelines • Integrate assessment, content and product data into shared enterprise data platforms • Ensure data quality, consistency and operational reliability 🤖 AI & GenAI Feature Enablement • Build and maintain data foundations powering LLM and GenAI capabilities • Design schemas and pipelines supporting AI-driven products and document processing • Handle complex parsed, normalised and sensitive datasets at scale • Support emerging AI use cases including evaluation processing, bulk uploads and intelligent workflows 🏥 IEP & Medicaid Data Integration • Build robust integrations across IEP, Medicaid and associated service platforms • Deliver reconciliation, data quality and cross-system consistency • Support enterprise reporting and operational analytics requirements 🔹 Essential Experience ✅ Deep expertise in Databricks and Delta Lake architecture ✅ Proven experience implementing and operating Medallion Architecture (Bronze, Silver, Gold) ✅ Advanced pipeline engineering experience including: • Orchestration • Idempotent processing • Error handling • Monitoring • Recovery strategies • Data reconciliation ✅ Strong Unity Catalog experience including: • Data governance • Lineage • Fine-grained permissions • Row-level and column-level security ✅ Extensive ETL and data integration expertise across complex enterprise systems ✅ Strong understanding of: • Data quality frameworks • Data normalisation • Backfilling strategies • Data integrity controls ✅ Experience handling sensitive and regulated data environments ✅ Knowledge of PII protection techniques including: • Masking • Tokenisation • Anonymisation ✅ Familiarity with GDPR and broader data privacy standards ✅ Performance and cost optimisation expertise including: • Z-Ordering • Liquid Clustering • Data Skipping • Query tuning • Shuffle optimisation • Cloud cost control ✅ 8+ years software engineering experience ✅ 3+ years operating at Senior/Lead Data Engineer or Databricks Architect level 🔹 Highly Desirable ⭐ Databricks AI capabilities • Vector Search • Model Serving • LLM Pipelines • Retrieval-Augmented Generation (RAG) ⭐ EdTech domain experience ⭐ Experience with industry standards such as: • Ed-Fi • OneRoster • CEDS • 1EdTech • High-impact programme combining Data Engineering, AI and Enterprise Architecture • Opportunity to shape the future data platform of a growing organisation • Significant ownership and influence from day one