▶️ LISTEN TO THIS ARTICLE

Agentic RAG: How AI Agents Are Rewriting Retrieval

A January 2025 survey paper from Singh et al. at Colorado State cataloged a shift in retrieval-augmented generation: instead of treating retrieval as a single preprocessing step, more systems are using agentic patterns that can plan, retrieve, evaluate, and try again. The paper maps four architectural patterns that help explain how teams are experimenting with retrieval in production. Those patterns matter because some RAG projects run into operational issues that one-shot retrieval does not address.

What Broke With Traditional RAG

The standard RAG pipeline has a clean, appealing logic. User asks a question. System searches a vector database. Top results get stuffed into a prompt. Model generates an answer. Ship it.

The problem is that this pipeline can make assumptions that break under real-world pressure. One retrieval pass may not be enough. Retrieved documents may not be relevant. Even when relevant documents are present, the generator may not ground its answer in them.

The RAG-E framework found, in its evaluation setting, that generators often ignore their own retriever's top-ranked documents. That is more than a minor accuracy issue: it means retrieved evidence is not guaranteed to control the answer. We covered this in detail in The RAG Reliability Gap.

The deeper issue is architectural. Traditional RAG often treats retrieval as a one-shot operation, something that happens before generation and then gets out of the way. But some real questions do not work like that. A question such as "What was Company X's revenue growth compared to competitors in Q3 2025?" can require multiple retrievals across different documents, with intermediate reasoning between each one. A single vector search is often too shallow for that kind of task.

One agent handles vector search over internal documents.

Enter the Agent

Agentic RAG flips the relationship between retrieval and generation. Instead of retrieval being a preprocessing step, it becomes a tool that an agent decides when and how to use. The agent can retrieve once, evaluate what it got, decide the results are garbage, reformulate the query, try again, pull from a different source, validate the new results, and then generate an answer. Or it can skip retrieval entirely if the question doesn't need it.

The survey on agentic RAG by Singh, Ehtesham, Kumar, and Talaei Khoei identifies four design patterns that give agents this flexibility: reflection, planning, tool use, and multi-agent collaboration. These are not just theoretical categories; they map onto patterns that are starting to appear in production-oriented systems.

The key difference from basic RAG architecture patterns is agency. In a traditional iterative RAG system, the pipeline is hardcoded: retrieve, check, retrieve again, generate. The loop is fixed. In an agentic system, the agent decides whether to loop at all. It decides which tools to use, how many retrieval passes to make, and when to stop. That decision-making capacity is what separates "iterative" from "agentic."

The Four Patterns

Single-Agent RAG

The simplest agentic pattern puts one agent in charge of the entire retrieval-generation pipeline. The agent has access to tools: vector search, web search, SQL queries, calculators, whatever the task requires. It receives a question, reasons about what information it needs, selects the appropriate tool, evaluates the result, and decides what to do next.

This is what most people mean when they say "agentic RAG." Frameworks like LangGraph make this pattern accessible by giving teams a structured way to run a ReAct-style loop (reason, act, observe) and course-correct mid-flight.

The strength is simplicity. One agent, one reasoning thread, full visibility into the decision chain. The weakness is bottlenecks. When the question requires expertise across multiple domains, a single agent can struggle to hold all the necessary context, exactly the goldfish brain problem applied to retrieval.

Multi-Agent RAG

The multi-agent pattern assigns specialized sub-agents to different retrieval tasks. A coordinator agent receives the query, decomposes it into sub-questions, and delegates each one to a specialist. One agent handles vector search over internal documents. Another runs SQL queries against structured data. A third searches the web for recent information. The coordinator aggregates the results and generates a unified answer.

This pattern shines when the question spans multiple data sources or requires different retrieval strategies for different parts of the answer. "Compare our Q3 revenue to industry benchmarks" needs internal financial data (structured query) and external market data (web search), and no single retrieval strategy handles both well.

The trade-off is coordination overhead. Every inter-agent message adds latency. Every delegation decision introduces a potential failure point. And debugging a multi-agent retrieval chain is significantly harder than debugging a single-agent loop. Sometimes the coordination tax exceeds the benefit, which is why single agents still beat swarms for many retrieval tasks.

Hierarchical RAG

Hierarchical agentic RAG adds explicit tiers to the multi-agent pattern. A top-level planner decomposes the task into high-level subtasks. Mid-level orchestrators manage groups of retrieval agents. Low-level executors handle individual search and extraction operations. Information flows up through the hierarchy, with each level performing validation and synthesis before passing results to the next.

The A-RAG framework from Du et al. demonstrates this approach with three hierarchical retrieval interfaces: keyword search, semantic search, and chunk read. The agent adaptively selects which granularity to use for each sub-query, retrieving across multiple levels before synthesizing. In the paper's benchmarks, A-RAG improved on flat approaches while using comparable or fewer retrieved tokens, which matters when you're paying per token.

Hierarchical patterns can outperform flat or single-agent baselines in some domain-specific and multimodal QA evaluations. That can be a meaningful gain, but the architecture is the most complex of the four patterns, and complexity has a cost that benchmarks do not always capture.

Adaptive RAG

Adaptive RAG is less a fixed architecture and more a routing strategy. The system classifies incoming queries by complexity and routes them to the appropriate retrieval pipeline. Simple factual questions get a single retrieval pass. Multi-hop reasoning questions trigger an iterative agentic loop. Questions that require real-time data get routed to web search agents.

The logic mirrors how human researchers work. You don't launch a full literature review to answer "What year was GPT-3 released?" But you also don't do a single Google search to answer "How has the transformer architecture influenced protein folding prediction methods?" The adaptive pattern applies the right amount of retrieval effort to each question.

Corrective RAG (CRAG), published by Yan et al., is a concrete implementation of this idea. CRAG uses a lightweight evaluator to score the relevance of retrieved documents and triggers different retrieval actions based on that score: keep the documents if they are good, supplement with web search if they are marginal, discard and re-retrieve if they are bad. In the paper's evaluations, CRAG reported gains over standard RAG on PopQA and PubHealth. Self-RAG takes this further by training the model itself to decide when to retrieve, generating special reflection tokens that control retrieval behavior at inference time. In its reported evaluations, Self-RAG also outperformed selected ChatGPT baselines on fact verification and biography generation tasks.

The risk is that teams build agentic retrieval pipelines without enough instrumentation to know whether they are actually working.

What Production Actually Looks Like

The paper-level results are encouraging. The production reality is harder.

In one finance-focused evaluation using 10-K filings and earnings reports, agent-driven retrieval performed better than a traditional RAG baseline after decomposing financial queries, validating intermediate results, and triggering additional retrievals when the first pass was incomplete. For financial QA, where the cost of a wrong answer can be high, that kind of workflow may justify added complexity.

But production deployments also report new failure modes that did not exist in simpler systems: retrieval loops, where the agent keeps searching without converging on an answer; incorrect retrieval decisions, where the agent uses the wrong tool for the job; and over-retrieval, where broken confidence calibration causes the agent to pull far more context than necessary, blowing through token budgets.

LangChain's State of Agent Engineering survey suggests many organizations still rely on base models, prompt engineering, and RAG rather than fine-tuning. That makes retrieval quality a major variable in production AI systems. The risk is that teams build agentic retrieval pipelines without enough instrumentation to know whether they are actually working.

The security surface also expands. Indirect prompt injection, already a concern with basic RAG, becomes more dangerous when an agent can autonomously decide to retrieve from external sources and then act on what it finds. A poisoned document in a vector database doesn't just corrupt a single answer. If the agent uses that answer as input for its next retrieval step, the corruption cascades.

The Evaluation Gap Nobody Talks About

Here's what frustrates me about the agentic RAG discourse: retrieval accuracy gets far more attention than retrieval decisions. An agent that retrieves five times when two would have sufficed burned three unnecessary API calls, added latency, and ate through tokens that a customer is paying for. But there is still no widely accepted metric for retrieval efficiency in agentic systems.

The closest thing is "retrieved tokens per correct answer," which A-RAG tracks and optimizes. But that's one framework out of dozens. Most production teams measure whether the final answer is correct and call it a day. They don't measure whether the agent took a sensible path to get there. That's like evaluating a delivery driver solely on whether the package arrived, ignoring that they drove 200 miles to deliver something across town.

This matters because agentic RAG's cost profile can be very different from traditional RAG. A naive pipeline has predictable costs: one embedding lookup, one generation call, done. An agentic pipeline's cost varies with the complexity of the question and the quality of the agent's decisions. A well-calibrated agent might make two retrieval calls. A poorly-calibrated one might make many more, retrieve a large amount of marginally relevant context, and produce an answer that is only slightly better than what a single retrieval pass would have given.

Teams deploying agentic RAG need three metrics that most evaluation frameworks don't provide: retrieval decision quality (did the agent choose the right tool?), retrieval efficiency (did the agent use the minimum retrieval steps needed?), and confidence calibration (does the agent's certainty about its answer correlate with actual correctness?). Without these, you're flying blind with an expensive autopilot.

Where Agents Actually Help (and Where They Don't)

Agentic RAG's real value is not in making every query more accurate. It is in handling queries where a fixed retrieval pipeline is too brittle.

Multi-hop questions that require information from multiple documents, synthesized across multiple reasoning steps, are where agents can earn their keep. Questions that require different retrieval modalities (vector search plus SQL plus web search) can benefit from an agent that selects the right tool. Questions where the first retrieval attempt fails may need an agentic loop, or at least a corrective retrieval step, rather than a single fixed pass.

For simple, single-hop factual questions, the overhead of an agent is often waste. A well-tuned traditional RAG pipeline may answer "What's our return policy?" faster, cheaper, and just as accurately as an agentic system. The adaptive pattern exists precisely because not every question deserves the full agentic treatment.

The honest assessment: agentic RAG solves real problems that cost real money in production. The context window vs. RAG debate hasn't been settled, and agentic RAG adds a third option to the mix. But it also introduces failure modes, latency, and cost that traditional RAG avoids. The teams getting this right aren't the ones who slapped an agent on every retrieval pipeline. They're the ones who instrumented their systems, measured where traditional RAG was actually failing, and applied agentic patterns surgically to those specific failure points.

Enterprise RAG failure rates will not drop to zero because teams add agents to the mix. They are more likely to improve when teams stop treating retrieval as a solved problem and start treating it as a system that needs monitoring, evaluation, and the same rigor applied to other critical infrastructure. Agents are a tool for that job. They are not a replacement for doing the job.

Related: How Agent Memory Got an Architecture

Sources

Research Papers:

Industry / Case Studies:

Related Swarm Signal Coverage: