Imagine a manufacturer reviewing maintenance reports. One question is straightforward: find examples of machines stopping because a replacement part was unavailable. Another asks how many distinct incidents had that cause during the last year.
An AI assistant might answer both with a polished paragraph and several citations. The second answer needs much more evidence. Finding useful examples does not establish that the system accounted for every incident.
We build RAG document search and AI document analysis systems. Choosing between a ranked search and a collection-wide review starts with what the answer has to prove.
A shortlist cannot establish the total
A typical retrieval workflow selects relevant passages and gives them to a model. Those passages can support an explanation or help someone locate a record. They are not automatically a complete list.
Microsoft’s vector-search documentation describes returning the nearest k results, where k is the requested result count. Even its exhaustive nearest-neighbor option searches for the closest k vectors. The word exhaustive describes the vector search, not a verified review of every document against your business question.
Increasing the result limit may surface more evidence. It still does not show that every relevant document was retrieved, or that several passages do not describe the same event.
A search service can also return a count of indexed matches. Microsoft’s documentation distinguishes that count from the number of results on a page. A query match count answers what matched the query in that index. It does not independently establish which records satisfy a more complicated business definition.
Define what one count represents
For our hypothetical manufacturer, one equipment stoppage might appear in a technician’s report, a shift log, and a follow-up email. Counting those files would count three records, not necessarily three incidents.
Before processing anything, write down the unit: files, incidents, machines, or something else. Decide which date controls inclusion and what evidence qualifies as a parts-related stoppage. A part being ordered after a failure does not necessarily mean its absence caused the downtime.
We recommend resolving these choices with someone who understands the records. If two reviewers interpret the rule differently, a larger model will not settle the underlying definition for the business.
Keep the rule alongside the result so someone can reproduce the same review later.
Establish the collection before analyzing it
Create an inventory of the approved sources and eligible records for the review period. Record where each item came from, its identifier, and the version examined. Reconcile that inventory with the source systems rather than assuming the search index contains everything.
Account for attachments, archived folders, inaccessible files, and duplicates. If a source is out of scope, say so. Restrict the review to material the team is authorized to access.
Choose a cutoff or snapshot for the report. A collection that changes halfway through processing makes the denominator harder to explain. Microsoft’s search guidance notes that counts can be approximate while documents are being added, updated, or deleted.
The result should describe the collection actually examined. It should not quietly become a claim about every record the business possesses.
Give each eligible record an outcome
For a collection-wide review, we recommend tracking every eligible item through processing. Use explicit outcomes such as:
- Matches the agreed definition.
- Does not match the agreed definition.
- Needs a person’s decision.
- Could not be processed.
- Not yet processed.
Keep a link to the relevant evidence and the reason for the classification. For an incident-level count, also record how several documents were grouped into one incident. Ambiguous groupings belong in review, not in an invisible deduplication step.
A failed file must not become a negative answer. A model’s negative classification must not become a claim that a person reviewed it. These are different states with different implications.
If an initial search or screening model excludes records before detailed review, preserve those exclusions and test them. A missed record at that stage never reaches the more capable model. Processing every selected candidate is not the same as accounting for the whole inventory.
Test what the system misses
Check a labeled sample that includes genuine matches, near misses, unusual wording, and difficult records. Inspect negative classifications as well as positive ones. Reviewing only the documents the system selected cannot reveal all the relevant documents it rejected.
Google’s Document AI evaluation guidance distinguishes precision from recall. Applied to this review, precision asks how many selected items really match; recall asks how many known matches the system found. A tidy list of correct examples can still have poor recall.
A successful test gives evidence about the tested material. It does not prove that the full collection contains no classification errors. Decide how much human review is needed based on what a missed incident would change.
Our guide to evaluating document extraction covers the related question of measuring correction work and exceptions.
Count the records, then explain the result
Once classifications and incident groupings are stored, calculate totals with ordinary database queries or a checked spreadsheet. PostgreSQL, for example, provides row-counting functions. Counting a field excludes null values, so report missing classifications and unresolved states separately.
Ask the language model to explain those calculated results with their evidence. Do not ask it to infer a total from whichever passages fit into an answer.
In a hypothetical review of 1,000 eligible incident records, a report might show 72 classified matches, 908 classified nonmatches, and 20 unresolved records. Those categories add up to the inventory. Calling the result simply “72 incidents” would hide the unfinished review and the possibility of classification errors.
Include the source scope, cutoff date, classification rule, review status, and unresolved count beside the headline number. A reader should be able to tell what the number includes before using it.
If the question only needs a few useful examples, a search assistant may be enough. If the decision depends on every item being accounted for, plan the inventory, classification, and review process first. Tell us what you need to count and where the records live, and we can help assess the right approach.