RAG Explained: How Retrieval-Augmented Generation Works (and How to Build It Right)

Retrieval-augmented generation (RAG) is the most practical way to make a large language model answer questions about your data: internal docs, product catalogs, support tickets, contracts. Instead of hoping the model memorized the right facts, you retrieve relevant passages at query time and put them into the prompt, so the model generates an answer grounded in that evidence.

That one-sentence description hides a lot of engineering. Most disappointing RAG systems don't fail because of the LLM. They fail because retrieval returned the wrong context. This guide covers how RAG works, how to build each stage well, and how to tell whether it's working.

#Key takeaways

• RAG = retrieve relevant context → add it to the prompt → generate a grounded answer.
• It's usually the right first choice when you need answers over private or frequently changing data. Fine-tuning solves a different problem (see fine-tuning vs RAG vs prompting).
• Retrieval quality is the bottleneck. Chunking, hybrid search and reranking matter more than which LLM you pick.
• Evaluate retrieval and generation separately, with a fixed test set.

#Why RAG exists

LLMs have three limitations that RAG addresses directly:

1. Knowledge cutoff — the model knows nothing after its training data ends.
2. No private knowledge — it never saw your Confluence, your database or your PDFs.
3. Hallucination — asked about things it doesn't know, it will often produce a plausible guess.

By supplying the relevant source text in the prompt, you turn an open-ended recall problem into a reading-comprehension problem, which LLMs are much better at. You also get citations: every answer can point to the chunks it came from.

#The RAG architecture, end to end

A production RAG system has two pipelines.

#1. Ingestion (offline)

textSources → Extract text → Clean → Chunk → Embed → Store (vector + keyword index + metadata)

#2. Query (online)

textUser question → (Rewrite) → Retrieve candidates → Rerank → Build prompt → LLM → Answer + citations

Let's walk through each stage.

#Ingestion: getting your data ready

#Extraction and cleaning

Garbage in, garbage out applies with full force. PDFs with multi-column layouts, tables, headers and footers produce messy text. Before anything else:

• Use a layout-aware parser for PDFs and keep tables as structured text (Markdown tables work well).
• Strip boilerplate: navigation, cookie banners, repeated footers.
• Preserve structure such as headings, section titles and list hierarchy. It becomes valuable metadata.

#Chunking

You can't embed a 200-page manual as one vector, and you can't put it all in a prompt. So documents get split into chunks. Chunking strategy has an outsized effect on quality.

Practical defaults that work for most text: structure-aware chunks of roughly 300–800 tokens with 10–15% overlap, and each chunk prefixed with its document title and section heading. That prefix is cheap and dramatically improves retrieval, because a chunk that says "It must be renewed annually" is useless without knowing what "it" is.

#Embedding and indexing

Each chunk is converted to a vector with an embedding model and stored in a vector index, alongside metadata like source, URL, date, author and access permissions. If you're choosing where to store vectors, read vector databases explained.

Also build a keyword index (BM25). You'll want it for hybrid search.

#Query time: retrieval that actually finds the answer

#Query rewriting

Users write short, ambiguous questions, and in a chat the question often depends on history ("what about the enterprise plan?"). A cheap LLM call can rewrite the question into a standalone search query first. For complex questions, you can generate multiple sub-queries and merge the results.

#Hybrid search

Vector search is great at meaning ("how do I cancel" ≈ "terminate subscription"), but weak at exact tokens such as product codes, error messages and names. Keyword search is the opposite. Hybrid search runs both and fuses the rankings, commonly with Reciprocal Rank Fusion:

pythondef reciprocal_rank_fusion(result_lists, k=60):
    scores = {}
    for results in result_lists:
        for rank, doc_id in enumerate(results):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)

candidates = reciprocal_rank_fusion([vector_hits, bm25_hits])[:50]

#Reranking

First-stage retrieval optimizes for speed across millions of chunks. A cross-encoder reranker reads the query and each candidate together and scores relevance much more accurately. The standard pattern: retrieve 30–100 candidates, rerank, keep the top 5–10. Reranking is often the single largest quality improvement you can make to an existing RAG system.

#Metadata filtering and permissions

Filter by metadata before or during retrieval: product version, region, document type, date range. Enforce access control here too. If a user can't open a document, its chunks must never reach the prompt. Treat this as a security boundary, not a nice-to-have.

#Generation: building the prompt

A solid RAG prompt:

textYou answer questions using only the provided sources.
If the sources don't contain the answer, say you don't know.
Cite sources as [1], [2] after each claim.

Sources:
[1] (Billing Guide › Refunds) Refunds are available within 30 days...
[2] (Billing Guide › Enterprise) Enterprise contracts are billed annually...

Question: Can an enterprise customer get a refund after 45 days?

Guidelines that hold up in production:

• Explicitly allow "I don't know." Without permission, models fill gaps.
• Order matters. Put the most relevant chunks first, and keep the total context focused rather than stuffing 50 chunks in.
• Ask for citations and verify them in code where you can: the cited IDs must exist in the provided sources.
• Use a structured output (JSON with answer and citations) if the UI needs to render sources.

#Common RAG failure modes

#Evaluating a RAG system

Evaluate the two halves independently:

Retrieval metrics on a labeled set of question → relevant-chunk pairs:

• Recall@k: did the right chunk appear in the top k?
• MRR / nDCG: how high did it rank?

Generation metrics on question → reference-answer pairs:

• Faithfulness: is every claim supported by the retrieved context?
• Answer relevance / correctness: does it actually answer the question?

Start with 50–200 real questions from users or support logs. Tools that use an LLM as a judge help scale this, but spot-check the judge against human labels. Our LLM evaluation guide covers building these test sets.

#Advanced patterns worth knowing

• Parent-document retrieval — match on small chunks, return the larger surrounding section to the LLM.
• Contextual chunk enrichment — use an LLM at ingestion time to write a one-line context for each chunk ("This chunk is from the 2025 refund policy, section on enterprise plans").
• Agentic RAG — let an AI agent decide when and what to search, iterating until it has enough evidence.
• GraphRAG — extract entities and relationships into a knowledge graph for questions that span many documents.
• Structured retrieval — for questions about numbers and records, generate SQL against a database instead of searching text.

#FAQ

#Is RAG better than fine-tuning?

They solve different problems. RAG supplies knowledge at query time and is ideal for facts that change or must be cited. Fine-tuning changes the model's behavior, style or format. Many production systems use RAG for knowledge and light fine-tuning, or just good prompting, for behavior.

#Do I need a vector database for RAG?

Not always. For small corpora (thousands of chunks), a vector extension in PostgreSQL such as pgvector, or even an in-memory index, is plenty. Dedicated vector databases earn their place at large scale or with heavy filtering and multi-tenancy needs.

#How many chunks should I put in the prompt?

Usually 3–10 after reranking. More context raises cost and latency and can distract the model. Measure answer quality at different values of k on your eval set rather than guessing.

#Does a long context window make RAG obsolete?

No. Long context helps when you already know which documents matter, but you still need retrieval to find them among thousands or millions. Sending everything every time is slow and expensive, and quality often drops with irrelevant context.