Lead Data Architect – AI & Cloud Infrastructure

Role: Lead Data Architect – AI & Cloud Infrastructure Location: Leeds, Manchester, Halifax-Hybrid model Employment Type: FTC/Contract Position Overview We are seeking a visionary Lead Data Architect to spearhead the evolution of our enterprise data platform. In this role, you will bridge the gap between traditional data engineering and cutting-edge artificial intelligence. You will not build AI models from scratch; instead, you will architect the scalable frameworks, high-performance pipelines, and secure storage systems that power our Generative AI (GenAI) and Predictive Machine Learning (ML) initiatives. The ideal candidate will design enterprise-grade Blueprints for Vector databases, RAG (Retrieval-Augmented Generation) infrastructure, and unified data lakes that ensure our AI assets are secure, governed, and highly available. Key Responsibilities 1. AI & Generative AI Infrastructure Design • Architect RAG Pipelines: Design scalable end-to-end Retrieval-Augmented Generation infrastructure to inject real-time enterprise context into Large Language Models (LLMs). • Vector Storage Management: Select, implement, and optimise enterprise vector databases (e.g., Pinecone, Milvus, pgvector) for high-performance embedding storage and semantic search. • Agentic AI Enablement: Build high-throughput, low-latency data loops required to support autonomous AI agents in production. 1. Core Data Architecture & MLOps Integration • Unified Data Foundations: Scale our modern data stack utilizing Lakehouse architectures (e.g., Delta Lake, Apache Iceberg) to handle both unstructured AI data and structured analytics. • Feature Engineering Infrastructure: Design and maintain enterprise Feature Stores (e.g., Feast, Tecton) to serve unified data features consistently across offline training and online real-time inference. • Streamline MLOps Pipelines: Partner with ML Engineers to integrate data pipelines seamlessly with lifecycle tracking frameworks like MLflow or Kubeflow. 1. AI Data Governance, Privacy & Quality • Data Lineage Automation: Implement comprehensive data lineage tracking to audit the source datasets used for AI training, fine-tuning, and prompt context. • Security & Compliance: Architect automated data masking, anonymisation, and PII-filtering pipelines to prevent sensitive data from leaking into foundational models. • AI Data Cataloguing: Curate metadata structures within platform catalogs (e.g., Collibra, Atlan) to explicitly map physical data assets to corresponding AI applications. Required Skills & Qualifications • Experience: Minimum of 7+ years of experience in data architecture, data engineering, or enterprise infrastructure design. • Cloud Mastery: Deep architectural expertise in at least one major cloud platform and its AI ecosystem: • AWS: SageMaker, Bedrock, Glue, Redshift. • Azure: Azure OpenAI Service, Azure Machine Learning, Synapse. • GCP: Vertex AI, BigQuery ML, Dataflow. • Advanced Data Modeling: Proven success modeling for both traditional relational/NoSQL analytical engines and high-dimensional vector spaces. • Data Pipeline Frameworks: Hands-on experience with streaming and batch tooling including Apache Spark, Kafka, Flink, and orchestration tools like Apache Airflow or Prefect. • Programming Literacy: Strong proficiency in Python, SQL, and database internals. Preferred Qualifications • Experience with unified data clouds such as Databricks or Snowflake. • Relevant cloud certifications (e.g., AWS Certified Data Engineer, Azure Solutions Architect, Google Cloud Professional Data Engineer). • Familiarity with Docker, Kubernetes, and infrastructure-as-code (Terraform).