A document assistant can give a good answer on launch day and a wrong answer a month later without any application code changing. Someone replaces a policy, moves a folder, or changes who can read a file. The assistant keeps answering from yesterday’s version.
That is the maintenance problem with retrieval-augmented generation, or RAG. The system searches a document collection and gives a language model relevant passages to answer from. Keeping it useful means maintaining that collection, the search, and the answer together.
We build RAG systems for organizations that need answers from their own material. Before putting one into daily use, we recommend agreeing on the following responsibilities.
Keep the document library current
Start with a clear source of truth. Decide which folder, document system, or approved upload process supplies the assistant. Give someone responsibility for deciding when material becomes current, superseded, or no longer appropriate to search.
Consider a hypothetical equipment distributor with an internal returns handbook. The operations manager replaces a policy that requires approval after ten days with one that requires approval after seven. A successful upload is only the beginning. The updated passage needs to become searchable, and the superseded passage needs to stop appearing as current guidance.
Keep a record of the source document, its version, and when it was processed. If historical versions remain useful, label them and distinguish historical questions from questions about today’s procedure. A date in the filename should not be the only thing separating an obsolete instruction from an active one.
Deletion deserves a test too. Removing a source file should trigger the intended removal of its searchable text and any derived copies, subject to the organization’s retention requirements. Check the result by searching, rather than assuming the original deletion reached every layer.
Notice ingestion failures before users do
An assistant can remain online while its document updates stop working. A connector can lose access, a scan can fail text extraction, or a processing job can leave part of a collection unfinished.
Track the last successful update for each source, outstanding failures, and documents waiting to be processed. Put those signals somewhere the person responsible will actually see them. A quiet failure queue is still a failure queue.
Check a sample of newly processed files as well. A PDF being accepted does not establish that its tables, headings, and page references survived extraction. If a document cannot be read reliably, flag it for correction instead of letting the assistant quietly work from incomplete text.
The right update frequency depends on the material. A frequently revised operations manual and a closed historical collection need different schedules. Agree on an acceptable delay and what happens when the system exceeds it.
Test retrieval and answers separately
Keep a small, representative set of questions with reviewed source passages and expected answers. Include questions the collection cannot answer, ambiguous requests, and cases involving conflicting versions.
Run those checks after meaningful changes to the documents, retrieval settings, prompts, or model. A useful evaluation distinguishes two failures: the system did not retrieve the passage needed to answer, or it retrieved the right passage but produced an unsupported or incomplete answer.
Those failures call for different fixes. Rewriting a prompt will not repair a missing document. Adding more documents will not necessarily correct an answer that misreads the evidence already available.
Check citations as part of the answer, including whether the cited passage supports the claim. A working link alone is insufficient. Keep expert human review for important decisions, and add confirmed user-reported failures to the test set.
We cover the initial build in our introduction to RAG in New England. After launch, the same evaluation discipline helps tell whether a change actually improved the system.
Treat access changes as part of maintenance
A shared document index does not automatically mean everyone should see every document. Retrieval needs to respect the permissions appropriate to the signed-in user before restricted passages reach the answer-generation step.
Test that boundary when people join, leave, or move between teams. Check what happens when a document becomes restricted after it was indexed. Permission information that was correct at import time can become stale too.
Include the surrounding product in that review. Saved conversations, answer caches, source previews, and diagnostic logs can contain information from documents. Decide who can read them and how long they should remain available.
Running the model on your own hardware does not remove these responsibilities. Local deployment changes where processing happens; internal access still needs deliberate controls.
Watch operating costs and response times
A maintenance budget should separate recurring infrastructure from usage-dependent work and the time people spend keeping the system useful. Depending on the design, those costs can include hosting or local hardware, document extraction, embedding new material, model requests, backups, and support.
Measure query volume, response time, failed requests, and the work triggered by each document update. Investigate changes in those measurements alongside answer quality. A cheaper configuration is a poor trade if people have to reopen every source and redo the answer themselves.
For example, importing a large archive and answering daily staff questions are different workloads. Keeping their costs separate makes it easier to explain a temporary indexing spike without mistaking it for normal monthly usage.
There is no useful universal monthly price for RAG maintenance. The document volume, update rate, deployment, access requirements, and expected support all affect the work. Define those before comparing proposals.
Make ownership and recovery explicit
Before launch, name the people responsible for document approval, technical operation, and reporting incorrect answers. They can be the same person in a small organization, but the responsibilities should be explicit.
Write down how to pause unreliable answers, report an incident, and restore service. Test backups and keep enough configuration history to understand what changed. For substantial retrieval changes, compare the proposed version with the working version before replacing it.
Recovery should preserve current access restrictions and document withdrawals. Restoring an old index without checking those changes can bring back information that should no longer be served.
A practical handoff should leave the team with an approved source list and an owner for each collection, a way to see failed updates and report bad answers, and reviewed test questions with a record of recent results.
The team also needs a documented access model, backup process, and recovery procedure. Agree on the boundary between routine maintenance and new feature work so that both sides understand what ongoing support covers.
Our AI consulting work includes ongoing support, and our RAG builds cover ingestion, retrieval, evaluation, and the interface. If you are planning a document assistant, tell us where the documents live and how they change. That is a useful place to start planning what the system will need after launch.