🎧 LISTEN TO THIS ARTICLE

In June 2023, a New York federal judge fined attorney Steven Schwartz $5,000 for submitting a brief stuffed with AI-generated case citations. The cases didn't exist. The courts they supposedly came from hadn't heard them. The judges named in the rulings were real, but the opinions attributed to them were fabricated by ChatGPT. That moment became shorthand for everything that can go wrong when AI enters legal practice without guardrails.

Two years later, the problem has scaled. More than 300 documented instances of AI-generated fake citations have appeared in court filings since mid-2023, with the rate accelerating to roughly two or three per day by late 2025. A California attorney was fined $10,000 after 21 of 23 quotations in an appellate brief turned out to be fabricated. Lawyers representing MyPillow CEO Mike Lindell were fined $3,000 each for a filing packed with nonexistent cases.

These aren't theoretical risks. They're career-ending mistakes that retrieval-augmented generation was supposed to prevent. The question for legal teams in 2026 isn't whether to use AI for research. It's how to build retrieval systems that hold up under adversarial scrutiny.

Most RAG systems are built for enterprise search. A sales rep asks about pricing history, the system pulls relevant documents, and the model synthesizes an answer. If the answer is slightly off, someone catches it in a meeting and nobody gets sanctioned by a federal judge.

Legal RAG operates under fundamentally different constraints.

Citation accuracy is binary. A case either exists or it doesn't. A statute either says what you claim it says or it doesn't. There's no "close enough." When a Stanford study tested leading legal AI tools in 2025, they found that even purpose-built legal RAG systems hallucinate between 17% and 33% of the time. That failure rate would be tolerable for internal knowledge bases. It's disqualifying for court filings.

Precedent is hierarchical. Not all legal sources carry equal weight. A Supreme Court opinion overrides a circuit court ruling, which overrides a district court opinion. Legal RAG needs to understand jurisdiction, binding vs. persuasive authority, and whether a case has been overturned, distinguished, or affirmed. Standard vector similarity search treats all documents as equal. That's a structural failure in legal contexts.

Everything faces adversarial review. Opposing counsel will verify every citation, challenge every characterization, and flag every error. This isn't like a customer-facing chatbot where mistakes cause inconvenience. The Air Canada chatbot case established that companies are liable for their AI's misrepresentations. In legal practice, the stakes are higher. Wrong citations can lead to sanctions, malpractice claims, and disbarment proceedings.

Privilege is non-negotiable. Attorney-client communications uploaded to a RAG system don't stop being privileged. If that data leaks into model training, influences responses for other clients, or becomes discoverable, the consequences extend far beyond the original case.

Even purpose-built legal RAG systems hallucinate between 17% and 33% of the time.

Three platforms dominate legal AI research, and understanding what they actually do (and don't do) matters for anyone building or evaluating legal RAG.

Harvey AI has scaled to over 700 global clients and reached an $8 billion valuation by December 2025. Harvey routes each request through a cascade of specialized LLMs, RAG systems that incorporate public and user-provided data, and reasoning models like o1 that reduce hallucination through chain-of-thought verification. Internal testing showed 97% lawyer preference over GPT-4 for case law research, with a roughly 40% reduction in review time. Harvey complies with SOC2 Type II, GDPR, and ISO 27001, and its Vault feature maintains zero data retention.

Thomson Reuters CoCounsel reached 1 million users across 107 countries by February 2026. The August 2025 relaunch as CoCounsel Legal introduced agentic workflows and deep research capabilities grounded in Westlaw and Practical Law content. It handles bulk document review of up to 10,000 documents and guided multi-step workflows for tasks like drafting complaints and conducting jurisdictional surveys. Pricing bundles with Westlaw at roughly $150-400 per month per seat, though enterprise contracts for large firms can approach $900,000 annually.

LexisNexis Lexis+ AI performed best in independent testing, answering 65% of queries accurately in the Stanford study. That's better than Westlaw's 42% accuracy rate but still means one in three queries produces unreliable output. LexisNexis has invested heavily in citation verification layers, but no system has achieved anything close to the "hallucination-free" marketing claims that both providers have made.

These tools are useful starting points for legal teams that don't want to build from scratch. But their limitations reveal why custom legal RAG architecture matters.

Building legal RAG that survives court scrutiny requires architectural decisions that general-purpose RAG systems don't make. The patterns that work borrow from agentic RAG approaches but add verification layers specific to legal requirements.

Multi-Stage Retrieval With Authority Ranking

Legal retrieval needs at minimum two stages. The first stage performs broad semantic search across the document corpus, pulling candidate documents based on conceptual relevance. The second stage re-ranks results by legal authority: jurisdiction, court level, recency, and whether the case remains good law.

This is where standard vector search falls short. Embedding models don't understand that a 2024 Supreme Court opinion on qualified immunity outranks a 2019 district court opinion from another circuit, even if the district court opinion is a closer semantic match to the query. Authority-aware re-ranking requires structured metadata (court, date, jurisdiction, subsequent history) layered on top of vector similarity.

Production systems handle this by maintaining parallel indexes. One index stores embeddings for semantic search. Another stores structured legal metadata for filtering and ranking. The retrieval pipeline queries both, intersects the results, and applies jurisdiction-aware scoring before anything reaches the generation model.

Citation Verification Loops

The single most important architectural choice for legal RAG is closing the loop between generation and verification. When the model produces a citation, a verification step must confirm that the cited source exists, says what the model claims it says, and hasn't been overruled or withdrawn.

The HalluGraph framework from recent research demonstrates one approach: building knowledge graphs from source documents and comparing them against claims in the generated output. Entity Grounding checks whether parties, citations, and dates in the response actually appear in retrieved documents. Relation Preservation verifies that the relationships the model asserts (plaintiff won, court held X) are supported by the source text.

In practice, most production legal RAG systems implement a simpler three-step verification:

  1. Existence check. Does the cited case, statute, or regulation exist in the corpus? If the system cites "Smith v. Jones, 547 U.S. 112 (2006)," does that reporter citation resolve to a real opinion?
  2. Content match. Does the source say what the model claims? Extract the relevant passage and compare it against the model's characterization. This catches the subtle form of hallucination where real cases are cited for propositions they don't actually support.
  3. Authority check. Is the source still good law? Has it been overruled, superseded, or distinguished in the relevant jurisdiction? Citator integration (Shepard's, KeyCite, or equivalent) is essential here.

If any check fails, the citation gets flagged or removed before reaching the attorney. This adds latency, typically 2-5 seconds per citation. That's a worthwhile trade-off when the alternative is a sanctions motion.

Confidence Scoring and Transparent Uncertainty

Legal RAG must tell users what it doesn't know. When the system retrieves documents with low relevance scores, or when the verification loop catches inconsistencies, the output should say so explicitly. "Based on the available documents, this appears to be the controlling authority, but I found conflicting language in [case X]" is more useful than a confident-sounding answer that papers over ambiguity.

Confidence scoring works at two levels. Document-level confidence reflects how well the retrieved sources match the query. Claim-level confidence reflects how well each specific assertion in the generated output is supported by the retrieved sources. Surfacing both scores gives attorneys the information they need to decide where to invest their own verification time.

This connects to broader challenges with AI agent hallucination. The hallucination problem isn't unique to legal, but the consequences of undetected hallucination are more severe.

Human-in-the-Loop as Architecture, Not Afterthought

Every legal RAG system should be designed around the assumption that a human attorney will review every output before it goes into a filing. This isn't a limitation. It's a design constraint that shapes the entire system.

The interface should present retrieved sources alongside generated text, with clear provenance links showing which source supports which claim. Attorneys shouldn't have to guess where the AI got its information. They should be able to click through to the original document, read the relevant passage in context, and make their own judgment about whether the citation supports the argument.

Systems that bury their sources or present outputs as polished prose are optimizing for the wrong thing. In legal contexts, transparency beats polish.

The Hallucination Problem in Detail

Citation accuracy is binary. A case either exists or it doesn't. There's no close enough.

The Stanford study on legal AI hallucinations deserves closer examination because it reveals patterns that anyone building legal RAG should understand.

Researchers tested Lexis+ AI, Westlaw AI-Assisted Research, and Ask Practical Law AI using preregistered queries across multiple legal domains. The key findings:

  • Lexis+ AI hallucinated on roughly 17% of queries, the lowest among tested systems
  • Westlaw AI-Assisted Research hallucinated at nearly double that rate, around 33%
  • All systems occasionally fabricated specific elements: case names, reporter citations, procedural histories, or holdings that didn't match the actual source

The hallucination patterns weren't random. Systems were more likely to hallucinate on edge cases, ambiguous queries, and questions spanning multiple legal domains. They performed best on straightforward, single-jurisdiction questions with clear answers.

This matters for architecture because it suggests that RAG reliability degrades predictably. Systems need stronger guardrails for complex queries, multi-jurisdictional research, and novel legal questions where the training data is thin. Adaptive retrieval strategies that invest more verification effort in harder queries can allocate resources where they're most needed.

The providers' marketing claims of "hallucination-free" legal research were directly contradicted by the Stanford results. That gap between marketing and reality is why independent evaluation matters and why firms building their own systems should benchmark ruthlessly against known-answer queries before deploying.

Privilege and Confidentiality

In legal contexts, transparency beats polish.

A federal court in the Southern District of New York ruled in United States v. Bradley Heppner that communications with publicly available AI platforms are not protected by attorney-client privilege. The reasoning was straightforward: there's no attorney-client relationship with ChatGPT, no reasonable expectation of confidentiality, and no protected legal advice being rendered.

This ruling doesn't mean all AI-assisted legal work loses privilege protections. But it draws a clear line: the architecture of your AI system determines whether privilege survives.

Data isolation is the foundation. Each client's documents must be stored in logically or physically separate indexes. A query for Client A's matter must never retrieve documents from Client B's matter, even if they're semantically similar. This is the digital equivalent of ethical walls, and it needs to be enforced at the infrastructure level, not just the application level.

On-premise and private cloud deployments preserve privilege arguments. When data never leaves your controlled infrastructure, the argument that confidentiality was maintained is much stronger. Enterprise legal AI platforms with contractual confidentiality protections and zero-retention policies offer a middle ground. Public consumer AI tools offer no protection at all.

Audit trails matter for privilege disputes. If opposing counsel challenges whether privilege was waived by running documents through an AI system, you need to demonstrate exactly what data the system accessed, what it stored, and what it didn't. Logging every retrieval query, every document accessed, and every output generated isn't optional. It's the evidence you'll need if privilege is contested.

Multi-tenant risks are real. RAG systems that embed multiple clients' documents in the same vector space create cross-contamination risks. Even if retrieval filters prevent Client A's queries from returning Client B's documents, the shared embedding space creates theoretical attack surfaces. For high-stakes litigation, dedicated single-tenant instances are the safer choice.

The NIST AI Risk Management Framework and the EU AI Act's high-risk system requirements, which take effect in August 2026, both emphasize data governance and transparency. Legal RAG systems built today should anticipate these requirements rather than retrofitting compliance later.

Getting Started: A Practical Roadmap

Building legal RAG from scratch isn't realistic for most firms. Here's a grounded approach to getting started, with realistic timelines and costs.

Phase 1: Evaluate Commercial Tools (2-4 Weeks)

Before building anything custom, test what exists. Request trials of Harvey, CoCounsel, and Lexis+ AI. Run them against queries where you already know the correct answers. Track hallucination rates, citation accuracy, and how well they handle your specific practice areas.

Create a benchmark set of 50-100 queries spanning your firm's primary jurisdictions and practice areas. Include easy questions (well-settled law), hard questions (circuit splits, evolving standards), and trick questions (asking about overruled cases or repealed statutes). Score each tool's outputs systematically.

Phase 2: Identify Gaps (1-2 Weeks)

Commercial tools will handle 60-80% of standard legal research competently. The remaining 20-40% reveals where custom RAG adds value. Common gaps include firm-specific knowledge bases (internal memos, prior work product), niche practice areas with thin coverage in commercial databases, and multi-jurisdictional research that requires synthesizing across legal systems.

Phase 3: Build Targeted RAG Layers (4-8 Weeks)

Don't try to replace Westlaw. Build supplementary RAG that covers your specific gaps. This typically means indexing your firm's internal document management system with legal-metadata-aware chunking, building retrieval pipelines that combine internal knowledge with commercial database results, and adding the citation verification loops described above.

For the technical foundation, the same RAG architecture patterns that work in general enterprise contexts apply here, with the addition of legal-specific re-ranking and verification layers.

Phase 4: Deploy With Guardrails (2-4 Weeks)

Start with a limited rollout to a small group of attorneys who understand the system's limitations. Require manual verification of all AI-generated citations for the first 90 days. Collect feedback systematically and track accuracy metrics against your benchmark set.

Expected costs for a mid-size firm: $50,000-150,000 for initial setup including commercial tool licenses, vector database infrastructure, and custom development. Ongoing costs of $5,000-15,000 monthly for compute, API access, and commercial database subscriptions.

Phase 5: Monitor and Iterate (Ongoing)

Legal RAG is not deploy-and-forget. Laws change. Cases get overruled. New jurisdictions adopt different standards. The system needs regular reindexing of new documents, periodic re-evaluation against updated benchmark queries, monitoring of hallucination rates across practice areas, and updates to citation verification databases.

FAQ

The 300+ documented instances of fake AI citations almost all trace back to consumer AI tools used without verification.

You can, but you shouldn't rely on it for citations. ChatGPT and similar general-purpose models regularly fabricate case names, reporter citations, and holdings. The 300+ documented instances of fake AI citations in court filings almost all trace back to consumer AI tools used without verification. If you use ChatGPT for brainstorming legal theories or identifying search terms, verify everything independently through a proper legal database before it goes into any document a court will see.

The Stanford study found that the best legal RAG system (Lexis+ AI) answered 65% of queries accurately, while the worst tested (Westlaw AI-Assisted Research) was accurate 42% of the time. Traditional Boolean search on Westlaw or LexisNexis is more labor-intensive but doesn't hallucinate. The practical answer: RAG speeds up the initial research phase, but it doesn't replace the attorney's obligation to verify every citation. Think of it as a research assistant that's fast but occasionally confident about things it made up.

Sanctions range from monetary fines ($3,000-$10,000 in recent cases) to referrals for bar disciplinary proceedings. Courts have consistently held that attorneys are responsible for the accuracy of their filings regardless of what tools they used. "The AI told me so" is not a defense. In the Mata v. Avianca case, the judge noted that the attorney's failure to verify AI-generated citations before filing them constituted a violation of the duty of candor to the tribunal.

It depends entirely on the system's architecture. Public consumer AI tools like ChatGPT do not protect privilege, as confirmed by the Heppner ruling. Enterprise platforms with contractual confidentiality guarantees, zero-retention policies, and data isolation provide a defensible basis for maintaining privilege. The safest approach is on-premise or private cloud deployment where client data never leaves your firm's controlled infrastructure. Whatever system you use, maintain detailed audit logs documenting what data was accessed and processed.

Sources