Data Platform Architect

Data Platform Architect Location: London Working Pattern: Hybrid – 2 days per week onsite Contract: 12 months Rate: Up to £550 per day Key Responsibilities • Define and maintain the enterprise Databricks reference architecture, platform principles and engineering guardrails. • Establish workload decision frameworks covering serverless compute, job clusters, interactive compute and SQL Warehouses. • Design and govern Unity Catalog, including catalogs, schemas, workspaces, storage credentials and external locations. • Define enterprise identity, entitlement, RBAC, service-principal and least-privilege access patterns. • Establish cluster policies, tagging standards, budget controls and platform cost governance. • Define reusable patterns for ingestion, transformation, orchestration, CI/CD, observability and data quality. • Lead migration from legacy platforms and classic compute to appropriate Databricks-native and serverless architectures. • Review solution designs, manage architectural risks and resolve complex performance, scalability and security issues. • Monitor platform adoption, utilisation, performance and cost, driving continuous optimisation. • Evaluate emerging Databricks capabilities and translate them into practical enterprise patterns. What We're Looking For • 10–20 years' experience in data/platform architecture, with strong hands-on experience architecting and operating enterprise Databricks environments. • Deep knowledge of Databricks, Unity Catalog, Delta Lake, Lakeflow, Workflows and SQL Warehouses. • Strong experience with identity, RBAC, data security, secrets management, networking and audit solutions. • Experience with platform automation, CI/CD, monitoring, cost governance and performance optimisation. • Strong knowledge of AWS, including IAM, networking, private connectivity, storage, encryption, secrets management and infrastructure as code. • Experience translating platform capabilities into enterprise standards and architecture decision frameworks. • Strong stakeholder management, technical leadership, architecture governance and communication skills. • Experience leading POCs, technology evaluations, architecture reviews and senior stakeholder presentations. • Experience working across multidisciplinary teams including data engineers, architects, governance specialists, analysts and business stakeholders. Desirable Skills • Databricks certification, such as Databricks Certified Data Engineer Associate/Professional or Solutions Architect. • Experience establishing or supporting a Databricks Centre of Excellence. • Terraform and Databricks Asset Bundles. • AWS CodePipeline, CodeBuild or GitHub Actions. • Python, SQL, Spark and REST APIs. • Real-time/event-driven data architectures. • Power BI, Tableau or other analytical platforms. • Data Mesh and data product architecture. • Exposure to AI/ML workloads on Databricks. • Experience working with globally distributed teams and enterprise-scale data platforms. Primary Technologies Databricks | PySpark | Delta Lake | Unity Catalog | Lakeflow/Delta Live Tables | SQL | AWS | Git | CI/CD | Data Modelling