Retrieval-Augmented Generation (RAG), Explained

Retrieval-augmented generation (RAG) is a technique that grounds a language model in external knowledge: when a question comes in, the system first retrieves the most relevant passages from your own data, then has the model generate its answer from those passages — cutting hallucinations and enabling citations without any retraining.

It fixes the model’s built-in limitation. A language model only knows what it learned during training; ask it about your company’s internal docs, last week’s release notes, or a customer’s order history and it will either admit ignorance or — worse — confidently make something up. RAG fetches the relevant information at query time instead of hoping the answer is baked into the weights.

It’s the difference between a closed-book exam and an open-book one. Same student, far better answers.

Retrieve, then generate

When a user asks something, the system searches a knowledge base for the most relevant passages, then inserts those passages into the prompt so the model can answer from them. The model still does the writing — it just isn’t working from memory alone anymore.

The payoff is threefold: answers reflect your current, proprietary data; the model can cite its sources, which builds trust and makes answers checkable; and you avoid the cost and rigidity of retraining the model every time your data changes.

The pipeline

RAG has two phases — one you run ahead of time, one that runs per query.

Ingestion (done once, updated as data changes):

1. Chunk your documents into passages small enough to be specific but large enough to be meaningful.
2. Embed each chunk — convert it into a vector (a list of numbers) that captures its meaning using an embedding model.
3. Store those vectors in a vector database so they can be searched by similarity.

Query time (every request):

1. Embed the user’s question with the same model.
2. Search the vector store for the chunks whose vectors are closest to the question’s — semantic similarity, not keyword matching.
3. Assemble the top results into the prompt, usually with instructions to answer only from the provided context and to cite it.
4. Generate the grounded answer.

The vector store is the engine room here. Many teams now use Postgres with the pgvector extension, keeping retrieval next to their existing data — part of the broader move toward edge and developer-friendly databases.

RAG vs. fine-tuning

A common question: why not just fine-tune the model on your data? Because the two solve different problems.

	RAG	Fine-tuning
Best for	Injecting knowledge that changes	Teaching behavior, tone, or format
Freshness	Update the index anytime	Requires retraining
Citations	Natural — you know the source	Hard — knowledge is diffuse
Cost to update	Low	High

In practice they’re complementary: fine-tune to shape how a model responds, use RAG to control what it knows. For knowledge that changes — docs, tickets, catalogs — RAG almost always wins.

The hard parts

RAG is easy to prototype and surprisingly hard to make excellent. The failure modes cluster around retrieval:

• Chunking. Split too small and you lose context; too large and you bury the relevant sentence in noise. There’s no universal setting — it depends on your content.
• Retrieval quality. If the right passage isn’t retrieved, the model can’t use it. Hybrid search (combining semantic vectors with keyword search) and a reranking step that re-scores candidates both help a lot.
• Evaluation. “It seems better” isn’t a metric. Measure retrieval accuracy and answer quality on a real test set, or you’re tuning blind.

How RAG is evolving

Two shifts are reshaping the pattern. First, massive context windows — many current models hold a million tokens — mean you can stuff far more retrieved material into a single prompt, relaxing the pressure on perfect chunking. Second, agentic RAG: rather than a fixed retrieve-then-generate pipeline, the model treats search as a tool it can call, iteratively, deciding what to look up and when. The Model Context Protocol makes that tool access standard, so an agent can query a knowledge base the same way it calls any other tool.

The takeaway

RAG is the most reliable way to make a general model speak accurately about your specific world — without retraining and with receipts. Start simple (chunk, embed, retrieve, cite), measure retrieval quality honestly, and add reranking or agentic retrieval only when the data shows you need it. It pairs naturally with running models locally for private, self-hosted setups where your data never leaves your infrastructure.