You remove an obsolete handbook from the shared drive. The next morning, the company AI assistant still quotes it. The original file is gone, but the answer is not.
That does not necessarily mean the model was trained on the handbook. A retrieval-augmented generation system, or RAG system, usually searches prepared copies of documents and supplies selected passages to a model. Removing the original and withdrawing those copies are separate operations.
We recommend making document withdrawal part of the acceptance test for a company RAG assistant. A useful demonstration starts with a document the system can find, removes it through the normal business process, and checks what remains available afterward.
Decide what “removed” means
First distinguish three requests. “Stop using this as current guidance” is different from “restrict it to the archive team,” and both differ from “delete the stored copies.” A superseded procedure might still be needed to explain an old decision without being suitable for today’s instructions.
For a hypothetical equipment rental company, imagine a retired delivery guide that says customers may collect equipment from a depot that has closed. This is an invented example, not a JTPCK client. The immediate requirement is to stop recommending that depot. Whether the guide remains in a historical archive is a separate decision for its owner.
Write that intended outcome before choosing deletion settings. Do not turn an operational cleanup into an indiscriminate purge. Have the responsible people settle retention obligations and approved exceptions, then give the implementer a clear rule to enforce.
Check whether the connector notices deletions
A connector that imports new and changed files may not notice when a file disappears. Microsoft’s Azure Storage indexing documentation explicitly distinguishes change detection from deletion detection. Its supported soft-delete approaches require configuration, and the indexer must observe the deletion while the relevant marker still exists.
The practical question is not “does it sync?” Ask what event tells the assistant to withdraw a document, how long that takes, and what happens if the connector is offline during that period. A successful import last week says nothing about today’s deletion.
For our rental-company example, we would ask the supplier to demonstrate the exact removal path staff use. Deleting a document manually in the search database proves that an administrator can remove it. It does not prove that deleting it in the shared drive produces the same result.
Follow the document’s identity, not just its name
We recommend keeping a stable source identifier and a version reference through processing. If the delivery guide produces several passages, an extracted table and a preview, the system needs a way to identify which records came from that guide.
A filename alone makes this harder. Someone might rename the guide, upload a replacement with the same name, or store a second copy in another folder. Ask whether removal applies to one version, one source record, or all authorized copies of the material. Those choices should be explicit.
OWASP’s RAG security guidance calls for deletion to propagate to chunks, embeddings, derived indexes and cached responses, and for checks that find orphaned chunks. In plain terms, the cleanup should follow the document through the system rather than stop at the upload screen.
Separate a request from a verified result
“Deletion requested” should not be displayed as “deletion complete.” Some search operations run in batches or asynchronously. Elastic’s delete-by-query documentation, for example, describes a starting snapshot, possible version conflicts and failures that can stop processing after earlier deletions have succeeded.
We would want a visible result that says which source and version were withdrawn, which processing stages finished, and which still need attention. An operator should be able to distinguish a completed withdrawal from a retry waiting in a queue.
For an urgent withdrawal, agree on a way to prevent affected material from being served while cleanup is incomplete. That is a containment measure, not proof that every stored copy has been erased. The distinction matters when explaining the result to staff.
Test ordinary use and the awkward follow-up
Use a synthetic guide with a distinctive depot name and a simple instruction found nowhere else in the test collection. Confirm that the assistant can retrieve it before withdrawal. Otherwise, an empty result afterward proves very little.
After removal, try the ordinary question, an exact phrase, the document title and any saved source-preview link. Inspect the retrieved passages as well as the final answer. A model that happens not to quote an obsolete passage has not demonstrated that retrieval stopped returning it.
Then reopen a conversation that used the guide earlier. Ask a follow-up without starting a new chat. The old text may already be in that conversation, even if fresh search no longer returns it. Decide how the product should label or handle that history instead of assuming a search-index cleanup rewrites prior conversations.
A screenshot or export already delivered to someone is another boundary. Do not promise to recall copies outside the system’s control. Our guide to document permissions covers the related distinction between future access and information already received.
Make withdrawal survive the next import
The most revealing test may happen later. Run the normal import again, including any job that was queued before withdrawal, and check that the retired guide does not return. In a test environment, repeat the check after the documented recovery procedure too.
We recommend recording withdrawals in a way that the import and recovery processes can consult before making old material available. The implementation will vary, but “it disappeared once” is a weaker result than “it stayed withdrawn through the next processing cycle.”
Backups and operational logs need their own documented treatment. Do not describe a clean search result as evidence that every backup has been erased. OWASP likewise separates handling of audit logs according to applicable retention and erasure requirements.
The handoff should leave the business with a repeatable test, an accountable operator and clear language for pending, blocked and completed withdrawals. Our RAG maintenance guide covers the wider operating plan. If you are evaluating an assistant, tell us how your documents are replaced and retired. Those everyday actions deserve as much attention as the first successful answer.