What Is Retrieval-Augmented Generation?
Retrieval-augmented generation (RAG) is a technique that grounds large language model outputs in external knowledge by retrieving relevant documents at inference time. Instead of relying solely on what the model memorized during training, RAG systems fetch context from a live corpus — databases, document stores, or web indexes — and feed it into the prompt alongside the user's query.
The approach matters because it addresses two persistent problems with LLMs: hallucination and stale knowledge. A model trained on data from 2024 cannot answer questions about events in 2026 unless you give it updated context. RAG provides that bridge without retraining the model.
RAG has become the default architecture for enterprise AI applications. From customer support bots to internal knowledge assistants, nearly every production LLM deployment uses some form of retrieval augmentation. But building a RAG system that actually works — one that retrieves the right context, formats it correctly, and produces faithful answers — remains a genuine engineering challenge.
Key Concepts
- Chunking strategies determine how source documents get split before indexing. Poor chunking is the most common cause of bad RAG outputs — too small and you lose context, too large and you dilute relevance.
- Embedding models convert text into vector representations for similarity search. The choice of embedding model often matters more than the choice of LLM for retrieval quality.
- Hybrid search combines dense vector retrieval with sparse keyword matching (like BM25) to capture both semantic similarity and exact term matches.
- Reranking applies a cross-encoder or LLM-based scorer to re-order retrieved chunks before they enter the prompt, improving precision at the cost of latency.
- Evaluation frameworks like RAGAS and TruLens measure retrieval relevance, answer faithfulness, and groundedness — critical metrics for production RAG systems.
Frequently Asked Questions
What is the difference between RAG and fine-tuning?
RAG retrieves external knowledge at query time without changing model weights, while fine-tuning modifies the model itself through additional training. RAG is better for frequently changing information and when you need source attribution. Fine-tuning is better for teaching the model new behaviors or styles.
How do you evaluate whether a RAG system is working?
Measure three things: retrieval precision (are the right chunks being fetched?), answer faithfulness (does the response accurately reflect the retrieved context?), and answer relevance (does the response actually address the question?). Frameworks like RAGAS automate these metrics.
What are the most common failure modes in RAG?
The top failures are: retrieving irrelevant chunks due to poor embeddings or chunking, exceeding the context window with too many retrieved documents, and the model ignoring retrieved context in favor of its own parametric knowledge. Each requires different mitigation strategies.