An enterprise RAG knowledge base is not “upload the whole drive into a vector database”. It is an engineering layer with four parts: curated sources, correct document chunking, hybrid retrieval with reranking, and permission enforcement. According to 2026 vendor reporting, between 70% and 80% of enterprise RAG projects never reach production, and most failures originate in the ingestion and retrieval layers rather than the model. This guide breaks down the four layers and offers a 90-day plan.
Why most RAG projects die before production
The pattern repeats itself. A team builds a pilot in two weeks, the demo is impressive, and then the system meets organisational reality and falls apart. A 2026 MLOps Community study of 143 enterprise RAG deployments found that 73% experienced at least one critical failure in their first quarter in production, and 41% of those failures were never flagged by standard evaluation suites.
The cause is almost never model selection. The widely cited industry estimate is that 80% of RAG failures trace back to the ingestion and chunking layer. In other words: when the system returns a wrong answer, in most cases the model was handed the wrong passage to begin with. That distinction matters because it changes where the budget goes – not into a model upgrade, but into retrieval engineering.
The budget implication: in 2026 RAG is not a model problem, it is a retrieval engineering problem. Improving chunking and reranking produces a bigger accuracy jump than moving to a more expensive model, at far lower running cost. If you are tracking the unit economics of an AI project, that is the difference between a variable cost that scales with usage and a one-off investment in infrastructure.
The four layers of an enterprise knowledge base
| Layer | What it does | Common failure |
|---|---|---|
| Sources and metadata | Which documents go in, who owns them, when they were last updated | Ingesting everything at once, including stale versions and drafts |
| Chunking | Splitting documents into the passages the model receives as context | One chunking rule for every document type, breaking tables and sections |
| Retrieval and ranking | Finding relevant passages and returning the best ones | Relying on vector search alone, with no keyword matching |
| Permissions | Ensuring a user only sees what they are allowed to see | Permissions checked in the interface rather than at retrieval time |
How to chunk documents properly
Chunking is the decision that most affects accuracy, and it is also the decision most teams make by default. Three families of methods are relevant in 2026:
| Method | Best suited to | Recommended chunk size |
|---|---|---|
| Fixed-size recursive character | Running prose, policy documents, process descriptions | 256-512 tokens for factual Q and A |
| Semantic chunking | Technical documents, specifications, product documentation | 512-1024 tokens for analytical tasks |
| Late chunking | Long documents where context crosses section boundaries | The full document is embedded first, then split |
A practical rule of thumb: a chunk should contain a complete answer to a single question. If an employee were handed that passage on paper, could they answer from it without looking anything else up? If not, the chunk is too short. If the passage covers three different subjects, it is too long and vector search will struggle to focus it.
Two recurring mistakes: breaking tables in the middle, which turns numeric data into noise, and losing the hierarchical heading. The fix for the second is cheap – prepend a context line to every chunk with the document name, the section and the date.
Hybrid retrieval and reranking
Vector search alone misses exactly what organisations ask about: part numbers, customer names, error codes, contract clauses. These are exact strings that semantic embedding blurs. That is why hybrid retrieval, combining dense embeddings with BM25-style keyword matching, continues to outperform dense-only search across most enterprise document types.
The structure that has become a common starting point: retrieve 20 candidates, rerank them with a reranker model, and pass the top five to the LLM. Reranking is the highest-return step relative to effort, because it corrects retrieval mistakes without touching the chunking layer.
Permissions: the layer everyone defers
This is where pilots die. In a pilot everyone can see everything, and then comes the moment when a support agent must not see payroll documents. If permissions are enforced only in the interface, retrieval still reads the document and the model may still quote from it. Enforcement has to happen at retrieval time, as a metadata filter, before passages ever reach the model.
In practice: every chunk carries the group identifiers permitted to see it, and the query filters on them. That is also what makes the system auditable, because you can show exactly which documents fed each answer.
How to measure that the system works
Roughly 70% of teams running RAG in production have no systematic measurement of retrieval quality. That is why failures go undetected. The accepted framework is the “RAG triad”, three axes you can run automatically against a fixed question set:
| Axis | The question it tests | What to do when it is low |
|---|---|---|
| Context relevance | Did retrieval find the right documents? | Fix chunking, add hybrid retrieval |
| Groundedness | Is the answer actually supported by the retrieved passages? | Tighten the prompt, require source citation |
| Answer relevance | Does the answer address what was asked? | Decompose multi-hop questions |
A known weak spot: 71% of production RAG systems fail on multi-hop questions, the kind that require combining information from two different documents. If that is your core use case, build the evaluation set around it from day one rather than around simple factual questions that always pass.
What separates teams that ship: an evaluation set that is documented, versioned and run automatically every night. Without it, every change to chunking or prompting is a gamble, and you cannot tell whether you improved things or broke them.
A 90-day plan
Days 1-30: pick one use case with a defined audience. Map only the 50 to 200 relevant documents, purge stale versions, and assign every document an owner and a last-updated date. In parallel, write 40 real questions the team actually asks, with approved answers. That is your evaluation set.
Days 31-60: build the pipeline – chunking, embedding, hybrid retrieval, reranking, permission filter. Run the evaluation set and record a baseline score on all three axes. Improve chunking and ranking until context relevance stabilises.
Days 61-90: open it to a real pilot group, log every question together with the passages retrieved, and add a feedback button. Failed questions become additions to the evaluation set. Only after the metrics hold steady for two consecutive weeks should you expand to more sources.
If you are building this for the first time, read how to measure visibility in AI answer engines as well – the same measurement discipline applies on the external side. And when you reach organisational rollout, our AI implementation practice starts from exactly this mapping exercise.
Frequently asked questions
How many documents do you need to start an enterprise RAG project?
Far fewer than people assume. A single use case is well covered by 50 to 200 clean, current documents. Ingesting tens of thousands of documents at the start lowers accuracy, because stale versions and drafts compete for the same retrieval slots.
What is the recommended chunk size?
Between 256 and 512 tokens for factual questions, and between 512 and 1024 tokens for analytical tasks. The practical rule: a chunk should contain a complete answer to a single question, without breaking tables and without losing the section heading.
Why is vector search alone not enough?
Because it blurs exact strings such as part numbers, error codes and customer names. Hybrid retrieval, combining embeddings with BM25-style keyword matching, produces better results across almost every enterprise document type.
How do you ensure a user never sees a document they are not allowed to see?
Enforce permissions at retrieval time rather than in the interface. Every chunk carries the group identifiers permitted to see it, and the query filters on them before any passage reaches the model.
How long does it take to reach production?
A single focused use case can reach a real pilot in about 90 days: one month to map sources and build an evaluation set, one month to build and tune the pipeline, and one month of piloting with real feedback.