A company handbook contains the right answer. The AI assistant finds a sentence from it. Yet the response misses the condition that changes what an employee should do.
Sometimes the missing information was lost when the document was divided into searchable passages. That step is called chunking. It sounds like a storage detail, but it determines which pieces of evidence travel together.
For a business owner evaluating a RAG system, the useful question is not “What is the best chunk size?” It is “Can the passages this system retrieves support the decisions our staff need to make?”
Start with a complete answer, not a word count
Consider a hypothetical equipment-rental business. Its handbook has a section on extending a rental, followed by a separate paragraph explaining what happens when someone else has reserved the equipment. These are invented circumstances, not a client project.
A passage containing only the extension procedure may look like a perfect match for “Can I keep this another day?” It is incomplete if the reservation condition never reaches the answer model.
Before choosing passage lengths, we recommend marking the smallest set of material needed to answer several real staff questions. Some answers fit in a paragraph. Others require a heading, an exception, or a second section.
Ask the person responsible for the handbook to identify those relationships. A developer can measure text length, but that alone will not establish which qualifications matter to the business.
Check that the text was right before it was split
Chunking cannot recover a table whose columns were already mixed up during PDF extraction. Nor can it recover a scanned page that never became readable text.
Our guide to checking PDF ingestion addresses that earlier step. Once the extracted material is reliable, inspect the actual passages produced from it, not just the original file.
For the rental example, place the handbook beside the indexed passages. Find the extension procedure, then locate the reservation condition. Are they separated? Does either passage identify which procedure it belongs to? This makes the proposed fix concrete.
Compare boundaries before adding complexity
Microsoft’s chunking guidance describes fixed-length passages, splits based on document structure, and combinations of the two. It also notes that a document can fit within a model’s input limit yet cover too many topics to be represented well as one searchable unit.
We recommend starting with a simple baseline and one alternative that addresses an observed failure. For a clearly organized handbook, that might mean comparing length-based splitting with sections that retain their headings.
Do not assume a heading makes a section self-contained. “Exceptions” is not useful context unless the passage also identifies the rule it qualifies. Conversely, a short FAQ answer may need no elaborate splitting at all.
Keep the original document and source location attached to each passage. A reviewer should be able to inspect what was included, what was omitted, and where the passage came from.
Use overlap for a reason
Overlap repeats some text across neighboring passages. Microsoft describes it as a way to preserve continuity, with the useful amount depending on the material.
We recommend testing whether that repetition fixes a specific boundary problem. If a condition follows immediately after a procedure, overlap might help. If it sits several sections away, repeating a few nearby sentences will not reliably bring it along.
Also inspect the final evidence packet. Several overlapping hits may contain nearly the same text. Five search results should not be mistaken for five independent pieces of support.
For our hypothetical handbook, record whether the reservation condition actually appears in the material supplied to the answer model. More passages or more repeated words are not the success criteria.
Add context without turning it into evidence
A passage that begins “This applies only during the winter schedule” needs a subject. The surrounding section may provide it, but the isolated text may not.
Anthropic’s Contextual Retrieval work describes adding a short, chunk-specific explanation before indexing. This is one possible response to missing context, not a requirement for every document collection.
We recommend trying existing document titles and section labels first when they resolve the ambiguity. If generated explanations are used, keep them distinguishable from the original text and check them against the source. A mistaken explanation can point search toward the wrong interpretation.
In the rental example, an indexing label might identify the procedure and equipment category. It should not invent an eligibility rule. Any answer about eligibility still needs support in the handbook itself.
Do not solve every miss with a larger passage
A larger passage can include the missing condition, but it can also bring along material unrelated to the question. Sending an entire handbook changes the problem rather than proving it solved.
The 2023 research paper Lost in the Middle found that performance on its tasks depended on where relevant information appeared in long inputs. Those results describe the models and tests studied, not every model available today. The practical lesson is to test use of the supplied context, rather than treating input capacity as proof of reliable reading.
We recommend comparing both retrieval and final answers. If the needed condition reaches the model but the answer still omits it, changing passage boundaries again may not address the remaining failure.
Choose with a small, inspectable comparison
Keep the source documents, questions, answer model, and scoring rules unchanged while comparing chunking approaches. Include straightforward questions, questions that cross a boundary, and questions requiring separate sections.
For each approach, record whether all necessary evidence arrived, whether the answer preserved its conditions, how much text was supplied, and how long the response took. Review failures individually rather than relying only on an average score.
Choose the simpler approach when the more complicated one does not produce a meaningful improvement. Save the examples that exposed bad boundaries so future indexing changes can be checked against them.
We build chunking, retrieval, evaluation, and interfaces as parts of the same system. If your assistant finds the right document but keeps losing the answer, tell us what staff are trying to find. Describe the failure without sending confidential material first. The next step is to locate the missing relationship, not guess a larger number for the chunk-size setting.