# RAG explained: how retrieval makes business AI reliable

Published 25 June 2026 by Spark · Technology, explained · 5 min read
Canonical: https://hellospark.co.uk/writing/rag-explained

> A general-purpose model knows a lot about the world and nothing about your firm. RAG fixes that. It feeds the model your own documents at the moment of the question, so answers are grounded in your sources and can be checked. Here is how it works and when you actually need it.

RAG is how you stop an AI making things up. A general-purpose model like Claude or GPT knows a great deal about the world and nothing at all about your firm. Ask it about your onboarding policy and it will guess, confidently, because guessing is what these models do when they have no source. Retrieval-augmented generation closes that gap. It finds the relevant passages from your own documents and hands them to the model at the moment of the question, so the answer is built from your material rather than the model’s memory. That one change is the difference between a party trick and a tool a regulated firm can put in front of staff.

## What the letters actually mean

Retrieval-augmented generation is three steps in a trench coat. Retrieval: search your documents for the passages most relevant to the question. Augmented: paste those passages into the prompt alongside the question. Generation: let the model write an answer using that supplied context. The model still does the writing. What changes is that it is now writing from evidence you chose, not from a hazy statistical recollection of the entire internet.

The retrieval step usually leans on something called a vector search. Your documents are chopped into chunks, each chunk is turned into a list of numbers that captures its meaning, and the same is done to the question. The system then finds the chunks whose meaning sits closest to the question, even when the wording is different. Ask about ‘proof of address’ and it will surface the paragraph about ‘residential verification documents’ without you having to match the exact phrase. That is the part keyword search never managed.

## Why it reduces hallucination

A model hallucinates when it is asked for something it does not know and has no way to say so. Left to its own devices it produces the most plausible-sounding answer, which is often wrong in ways that read perfectly. RAG removes the vacuum. Instead of ‘what do you know about our fee policy’, the model is effectively asked ‘here are the three relevant paragraphs from our fee policy, now answer’. The scope narrows, the source is present, and a well-instructed system will say ‘that is not covered in the documents provided’ rather than invent. You never get to zero. You get from unacceptable to manageable, and for most business work that is the whole game.

## Why citations matter more than the answer

The quiet superpower of RAG is that it can show its working. Because the answer was built from specific retrieved passages, the system can tell you which document and which section each claim came from. For a regulated firm that is not a nice-to-have. It is the difference between an output a supervisor can inspect and a black box nobody can defend. A staff member reading a RAG answer can click through to the source clause and confirm it in seconds, which means the human check stays real rather than becoming a rubber stamp. We think citations should be non-negotiable in any system that touches client work, and we build them in from day one.

## When your firm actually needs it

You need RAG when the right answer lives in your own material and changes over time: policies, procedures, contracts, past correspondence, a knowledge base, a filing cabinet of PDFs. The classic cases in our world are a staff assistant that answers ‘how do we handle this’ from the firm’s own procedures, a tool that pulls the relevant clauses out of a long agreement, or a search layer over years of client files. If the useful knowledge is yours and specific, RAG is usually the honest answer.

You do not need it for everything. Summarising a document you have already pasted in needs no retrieval, because the source is right there. Drafting from a template, translating, reformatting, tone-fixing: none of these are RAG problems. And RAG is not a substitute for structured data. If the question is ‘what is the NAV’, that belongs in a database query, not a document search. Reaching for RAG when a simple lookup would do is a common and expensive mistake.

## Where it goes wrong

RAG is only as good as what it retrieves. Feed it messy, duplicated, out-of-date documents and it will faithfully ground its answers in your mess. Garbage in, confidently cited garbage out. The unglamorous work is in the retrieval: chunking documents sensibly, keeping the source current, and testing that the right passages actually surface for real questions. Most RAG projects that disappoint were under-built on that side and over-built on the chatbot. A related discipline is getting clean data out of the documents in the first place, which we cover in [AI document extraction](/writing/ai-document-extraction).

## How we build it

We treat RAG as plumbing, not magic. Scope the smallest set of documents that answers a real, repeated question. Get retrieval accurate before worrying about the wording of answers. Insist on citations. Keep a human in the loop wherever the output could reach a client or a regulator. Models such as Claude handle the generation step well because they follow the instruction to stay inside the supplied sources and to admit when something is not there, which is exactly the behaviour you want in regulated work.

If you have a shelf of documents that staff keep asking about, that is a RAG-shaped problem and a good first build. [Book a review](/book-a-review) for a candid read on whether it is worth doing, where retrieval would earn its keep and where a simpler tool would do the job.
