RAG for Enterprise Knowledge Bases: What Actually Breaks in Production

Retrieval-augmented generation looks deceptively simple on a whiteboard: chunk the documents, embed them, store the vectors, retrieve the relevant chunks at query time, hand them to an LLM. Every RAG demo we've seen works on the first ten questions someone tries. The failures show up three weeks after launch, when real employees ask real questions against a document set that's messier and more contradictory than anyone accounted for during the build.

Chunking decisions made in week one cause problems in month three

Chunk size and boundary choices get made early, almost as an implementation detail, and then quietly determine retrieval quality for the life of the system. Fixed-size chunking (say, 500 tokens with 50-token overlap) is easy to implement and wrong for a lot of enterprise content. A policy document with numbered clauses, a table of pricing tiers, and a legal contract with cross-referencing sections all need different chunking strategies — cut a table in half and neither chunk is useful to a retriever.

What we've found actually holds up:

• Structure-aware chunking that respects document boundaries — headings, list items, table rows — rather than blind token counts. This requires more upfront parsing work per document type but pays off directly in retrieval precision.
• Metadata-rich chunks that carry the source document title, section heading, and date alongside the text, so the retriever and the generation step both have context beyond the raw chunk content.
• Re-chunking as a planned maintenance task, not a one-time setup step. When document formats change or a new document category is added, chunking strategy needs revisiting — teams that treat chunking as "done" in week one see retrieval quality degrade as the corpus grows.

Stale and contradictory documents poison retrieval quietly

Enterprise knowledge bases are never a clean, current, non-contradictory set of documents. There's the 2022 policy that was never removed, the draft that got emailed around and somehow ended up indexed, and three versions of an onboarding guide with conflicting instructions. A vector search doesn't know which one is authoritative — it returns what's semantically similar, and an outdated document can be just as semantically similar to a query as the current one.

This is the single most common cause of "the AI gave a wrong answer" complaints we see post-launch, and it's rarely a model problem. Fixes that actually address the root cause:

1. Source-of-truth tagging — an explicit field marking which documents are authoritative, with retrieval configured to prefer or exclusively use tagged sources for high-stakes queries (compliance, HR policy, pricing).
2. A document lifecycle process — someone owns retiring outdated documents from the index, not just adding new ones. Without an owner, the index only grows and staleness compounds.
3. Recency weighting in the retrieval score, not just semantic similarity, so a 2025 document naturally outranks a 2022 one on similar content when both are retrieved.

Retrieval can succeed while the answer still fails

A subtle failure mode: the retriever finds exactly the right chunks, but the generated answer is still wrong, because the LLM either ignored the retrieved context in favor of its training data, or synthesized across chunks in a way that introduced an error not present in either source chunk individually. This is much harder to catch than a retrieval miss because the system logs look correct — the right context was passed in.

Guardrails that help:

• Explicit prompting that instructs the model to answer only from provided context and to say "I don't have information on this" rather than filling gaps from general knowledge — and testing that this instruction actually holds under adversarial rephrasing.
• Citation requirements, where the model must reference which retrieved chunk supports each claim. This doesn't just help end users trust the answer — it makes wrong answers easier to debug, because you can see whether the failure was retrieval or generation.
• A held-out set of "trap" questions where the correct answer is "we don't have that information," used specifically to catch hallucination rather than testing retrieval accuracy.

Access control is a harder problem than most teams expect

The question we ask every client early is: does every user asking a question have permission to see every document that might get retrieved for it? In a general-purpose company wiki, usually yes. In an enterprise knowledge base spanning HR, legal, and engineering documentation, almost never. A RAG system that ignores document-level permissions and just retrieves the most semantically relevant chunk is a data leak waiting to happen — an employee asking a general question could get an answer synthesized from a document they were never supposed to see.

Handling this correctly means permission-aware retrieval — filtering candidate chunks by the requesting user's access rights before they ever reach the LLM, not after. Retrofitting this onto a system that was built without it is expensive, which is why we push clients to define the access model before the first line of retrieval code is written, not after a security review flags it post-launch.

RAG is genuinely useful for enterprise knowledge work — the failure modes above are why the systems that stay useful are the ones built with document lifecycle management, access control, and evaluation baked in from the start, not layered on after the pilot succeeds.