Start with what the model needs to know
A model can be very capable and still lack your latest policy, internal documentation or a project detail that was never part of its training. RAG supplies selected external information when the question arrives. The goal is useful context, not a vector database for its own sake.
In 2026 I would still choose retrieval for a changing document collection with source tracking and access boundaries. For a small, stable set of documents, placing the relevant material directly in the prompt may be simpler. The right choice depends on the information problem and the measured user experience.
Long context and retrieval solve overlapping problems
A larger context window lets you include more material. Retrieval helps decide which material belongs in this request. Neither approach guarantees that an answer is grounded, current or permitted for the user.
Suppose a company has different refund rules by plan, region and policy version. Sending every rule leaves the model to resolve conflicting context. Retrieving the rule with the right metadata gives the answer a cleaner starting point. For a short document with no access distinctions, that extra pipeline may not earn its complexity.
Read the diagram as text
Long context and retrieval solve overlapping problems — A larger context window lets you include more material. Retrieval helps decide which material belongs in this request. Neither approach guarantees that an answer is grounded, current or permitted for the user.
What makes retrieval worth building
Preserve document identity, version and meaningful passage boundaries. A clause separated from its exception can be misleading even when similarity search returns it. Apply access rules before the model receives context, not after the answer is generated.
Anthropic’s Contextual Retrieval work illustrates two useful ideas: precise lexical matching can complement embeddings, and a passage may need surrounding context to remain interpretable. Reranking can refine candidates, but it adds cost and latency. Treat these as options to test against observed failures rather than a mandatory stack.
Primary sourceEvaluate the evidence before polishing the answer
Make a small evaluation set with questions you can answer by inspecting the documents. Include a paraphrase, an exact identifier, a contradictory old version and a question whose answer is absent. Inspect whether the necessary passage appears, then judge whether the response uses it correctly.
If retrieval misses the clause, a more persuasive prompt is unlikely to repair the knowledge gap. If retrieval succeeds but the answer ignores a condition, work on the answer stage. Keep these failures separate so you know what to improve. A grounded refusal is a useful outcome for an unanswerable question.
The boundary matters as much as the pipeline
Casual chat does not need retrieval from a personal portfolio. A greeting should not fail because no document contains “hello.” A factual question about a project does need the relevant records. Routing is part of the experience, not just an optimization.
My answer to the title is yes, when external evidence is the reason the application exists. Build the smallest retrieval system that can reliably find that evidence, measure it and keep uncertainty visible. RAG remains an engineering choice, not a universal badge of sophistication.
The sources behind the ideas.
Research and primary documentation for the factual points above. Examples and recommendations are my interpretation.
