Agentic RAG Engineer/Architect - London (Contract)

Agentic RAG Engineer / Architect Department: Global Markets — AI & Intelligent Automation Location: London City, Hybrid - Contract (Outside IR35) About the Team My client's Global Markets Division are building an enterprise-grade agentic platform that will transform how trading, sales, risk, and operations teams access, reason over, and act on information. A critical pillar of this platform is an agentic Retrieval-Augmented Generation (RAG) layer — not a simple "search and stuff" pipeline, but an intelligent, self-directing retrieval system that can plan queries, route across heterogeneous data sources, assess and refine its own results, and feed high-fidelity context to downstream agents. The data landscape is complex: unstructured research, regulatory documents, structured trade and risk data in relational and tabular stores, graph-based entity relationships, real-time market data, and internal knowledge bases. The RAG layer must unify all of this behind a coherent, agentic retrieval interface. Role Purpose They are hiring an engineer-architect who will own the design and build of the agentic RAG layer for Global Markets. You will architect a retrieval system where agents autonomously determine what data they need, plan retrieval strategies across vector stores, knowledge graphs, and tabular databases, evaluate the quality and relevance of retrieved context, and iteratively refine until the answer meets a quality threshold — all with full traceability and auditability. Key Responsibilities 1. Agentic RAG Architecture — Design an end-to-end agentic retrieval architecture where retrieval is not a static pipeline but an agent-driven process: query planning, source selection, parallel retrieval, result fusion, self-critique, and iterative refinement. Implement advanced patterns including corrective RAG (detect and recover from poor retrieval), adaptive RAG (dynamically adjust strategy based on query complexity), and self-reflective RAG (critique and refine answers before returning). 1. Knowledge Graph Integration — Build and maintain graph-based knowledge representations (entities, relationships, hierarchies) for Global Markets domains (counterparties, instruments, legal entities, regulatory taxonomies). Implement graph-RAG patterns for multi-hop reasoning and relationship-aware retrieval. 1. Vector Database Engineering — Architect and optimise the vector storage layer — embedding model selection, chunking strategies (semantic, hierarchical, sliding window), index tuning (HNSW, IVF-PQ), metadata filtering, hybrid search (dense + sparse), and re-ranking pipelines. 1. Structured Data Retrieval — Design agentic text-to-SQL and text-to-query capabilities for structured/tabular data stores, enabling agents to autonomously formulate queries against trade, risk, P&L, and reference data in relational databases and Databricks lakehouse tables. 1. Databricks Integration — Leverage Databricks as a core data platform — Unity Catalog for data governance, Delta Lake for reliable storage, Databricks Vector Search for managed embeddings, and Spark/SQL for large-scale data processing and feature engineering. 1. Query Routing & Orchestration — Implement intelligent query routing agents that decompose complex user questions into sub-queries, determine the optimal retrieval source (vector, graph, tabular, API) for each, execute in parallel, and synthesise coherent answers. 1. Retrieval Quality & Evaluation — Build evaluation frameworks for retrieval quality: precision, recall, MRR, faithfulness, answer relevance, and hallucination detection. Implement online and offline evaluation loops, including LLM-as-judge and human-in-the-loop feedback. 1. Ingestion Pipelines — Design and build robust data ingestion pipelines: document parsing (PDFs, HTML, spreadsheets), intelligent chunking, metadata extraction, entity resolution, embedding generation, and incremental index updates. Handle Global Markets document types: term sheets, trade confirmations, risk reports, regulatory filings, research notes. 1. Guardrails & Data Governance — Enforce access control at the retrieval level — ensuring agents can only retrieve data the requesting user is authorised to see. Implement PII detection, data classification, and audit trails for every retrieval operation. 1. Observability & Performance — Instrument the RAG layer with comprehensive tracing: query decomposition traces, retrieval latency per source, relevance scores, token usage, cache hit rates. Optimise for sub-second retrieval on hot paths. Required Skills & ExperienceRAG Engineering (must-have) ●       6+ years of software engineering experience, with at least 2-3 years focused on information retrieval, search systems, NLP, or RAG architectures. ●       Deep expertise in vector databases — production experience with at least one of: Pinecone, Weaviate, Milvus, Qdrant, Chroma, pgvector, or Databricks Vector Search. Understanding of index types, distance metrics, sharding, and performance tuning. ●       Hands-on experience with embedding models — OpenAI embeddings, Cohere, sentence-transformers, BGE, or similar. Understanding of embedding dimensionality, fine-tuning, domain adaptation, and cross-encoder re-ranking. ●       Production experience with chunking and ingestion pipelines — document parsing (unstructured.io, LlamaParse, Apache Tika), semantic chunking, contextual chunking (prepending document-level context to chunks), parent-child relationships, and metadata enrichment. ●       Experience with hybrid search — combining dense vector retrieval with sparse retrieval (BM25, SPLADE) and implementing re-ranking (Cohere Rerank, cross-encoders, ColBERT). Graph & Knowledge Representation (must-have) ●       Experience with knowledge graphs — design, construction, and querying of graph structures for entity-rich domains. Production use of Neo4j, Amazon Neptune, or TigerGraph (or equivalent). ●       Familiarity with graph-RAG patterns — using graph traversal, community detection, or subgraph retrieval to provide relational context to LLMs. ●       Understanding of ontology design and entity resolution for financial entities (counterparties, instruments, legal entities). Structured Data & Databricks (must-have) ●       Production experience with Databricks — Delta Lake, Unity Catalog, Spark SQL, and ideally Databricks Vector Search or Databricks Feature Serving. ●       Experience with text-to-SQL approaches — using LLMs to generate and validate SQL/Spark SQL queries against structured data, with schema-aware prompting and result validation. ●       Solid understanding of relational data modelling for financial data — trade lifecycle, risk measures, reference data, and P&L. Agentic Patterns (must-have) ●       Experience building agentic RAG systems — where retrieval is driven by autonomous agents that plan, route, execute, evaluate, and refine (not static pipeline RAG). Familiarity with patterns such as CRAG (Corrective RAG), Self-RAG, and Adaptive RAG. ●       Proficiency with LangChain / LangGraph for building retrieval agent graphs, tool-augmented retrieval, and self-reflective RAG loops. ●       Strong Python skills — async programming, type safety (Pydantic), and building production services. AI / LLM Fundamentals (must-have) ●       Deep understanding of LLM capabilities — context windows, function calling, structured output, prompt engineering, and model selection trade-offs. ●       Experience with RAG evaluation methodologies — RAGAS, custom evaluation pipelines, LLM-as-judge, and human annotation workflows. Financial Services (preferred) ●       Prior experience in capital markets technology — understanding of the data landscape (trade systems, risk platforms, market data, regulatory reporting). ●       Familiarity with data governance in banking — entitlements, data classification, regulatory data lineage (BCBS 239, GDPR). Preferred / Bonus Skills ●       Experience with multi-modal RAG — retrieving and reasoning over charts, tables, images, and structured data alongside text. ●       Familiarity with GraphRAG (Microsoft) or similar community-based graph retrieval approaches. ●       Experience with Apache Kafka or streaming pipelines for near-real-time document ingestion and index updates. ●       Knowledge of semantic caching strategies for reducing retrieval latency and LLM costs. ●       Contributions to open-source RAG or retrieval projects (LlamaIndex, LangChain retrievers, Haystack, etc.). ●       Experience with fine-tuning embedding models on domain-specific corpora (financial text, regulatory language). ●       Familiarity with data mesh / data product architectures in the context of serving retrieval use cases. Technical Stack ● Retrieval Orchestration LangGraph - LangChain, Pydantic.AI ● Vector Databases - Databricks Vector Search, Pinecone / Weaviate / Milvus / pgvector ● Knowledge Graphs - Neo4j, Amazon Neptune, or TigerGraph ● Data Platform - Databricks (Delta Lake, Unity Catalog, Spark SQL, MLflow) ● Embedding models - OpenAI, Cohere, sentence-transformers, BGE ● Re-ranking - Cohere Rerank, cross-encoders, ColBERT ● Document processing - Unstructured.io, LlamaParse, Apache Tika ● LLM providers - OpenAI (GPT-4+), Anthropic (Claude), Azure OpenAI ● Languages - Python, Rust ● Evaluation - RAGAS, custom evaluation harnesses, LangSmith ● Observability - OpenTelemetry, LangSmith / LangFuse, Grafana ● Infrastructure - Kubernetes (AKS) Qualifications ●       Bachelor's or Master's degree in Computer Science, Information Retrieval, Computational Linguistics, Data Science, or a related quantitative discipline. PhD in a relevant field (IR, NLP, knowledge graphs) is a strong differentiator. ●       Relevant certifications are valued (Databricks Certified, AWS/Azure ML Specialty) but are secondary to demonstrated hands-on expertise. Agents notes: This role requires in-depth knowledge and experience. I will reply to every applicant. Thanks.