A new model can write a better-looking answer and still be a worse fit for the software around it. A longer explanation might break a field intended for a short label. A more decisive answer might remove the uncertainty that used to send a difficult case to a person.
We recommend treating a model replacement as a software release, not a settings change. The question is whether the proposed version improves the complete workflow without losing behavior the business already relies on. That applies to a hosted model, a local model, or a new version of a fine-tuned model.
Our AI integrations connect models to business software. Those connections are exactly where a small change in output can become a larger operational problem.
Write down what is being replaced
Before testing, save the working configuration. Include the model identifier, prompt, output format, tool definitions, retrieval settings, and any application rules that interpret the answer. Keep an approved example of the input and output at each boundary.
Then describe the proposed change in one sentence. Replacing the answer model is different from replacing the answer model, rewriting the prompt, and rebuilding the document index at the same time. Start with the smallest useful comparison so a failure has somewhere to be traced.
If the new model needs different settings, record them rather than pretending the comparison changed only a name. We would test that whole candidate configuration as a release. A record of what actually ran is more useful than a claim that everything else was approximately the same.
Compare failures, not just average scores
Anthropic’s evaluation guidance recommends specific, measurable criteria tied to the task, including edge cases and comparisons with a baseline. Apply that to the decision your team needs to make.
Consider a hypothetical supplier using AI to turn incoming requests into draft order records. The existing system leaves an unclear quantity unresolved. The candidate fills in a plausible number. A reviewer might prefer the candidate’s tidy summary while missing that it guessed a field the old version correctly left blank.
We would record which cases both versions pass, which only the candidate passes, and which the candidate newly fails. That last group deserves explicit review even when the overall result improves. For this invented supplier, preserving uncertainty about quantity would be a release requirement, not a minor deduction in a combined score.
Keep a separate final test set that has not been used to rewrite the candidate’s prompts. Include ordinary work as well as troublesome examples. Repeat important cases to expose inconsistent behavior, and keep the actual outputs alongside the scores.
Test the application contract
The model’s answer must still fit the receiving software. Promptfoo’s assertion documentation describes checks for exact values, JSON structure, custom conditions, cost, and latency. Those are useful building blocks for verifying a release, rather than relying only on someone reading sample responses.
For the hypothetical order workflow, we would check allowed units, missing values, item identifiers, and whether the proposed record can be displayed and corrected in the existing interface. A field that parses successfully is not necessarily the right field value. Our guide to validating structured AI data explains that distinction.
Tool use needs its own test. Confirm which action the candidate requests, which record it targets, and what happens when required information is absent. Use a test environment or simulated tools so comparing versions cannot place two orders or send two replies.
An upgrade should not quietly widen the system’s authority. Keep existing application permissions and human approval requirements in force while evaluating the new model.
Decide what would stop the release
Write the stop conditions before looking at the results. For our invented supplier, an unauthorized order submission would stop the rollout immediately. A slower draft might be acceptable within an agreed limit. Those are different kinds of judgment and should not disappear into the same average.
Name the person who can pause the rollout and the place where staff report unexpected behavior. Give reviewers a way to identify which version produced a result. Otherwise, a complaint about “the assistant” is hard to connect to a specific change.
We also recommend checking the operational assumptions separately: whether the candidate uses a different provider, where data is processed, which features retain content, and whether the existing approvals cover that setup. Do not send real customer records to a new destination merely to complete a benchmark.
Limit the first live exposure
Google’s SRE guidance on canary releases describes evaluating a change on part of a service before expanding it. It also explains why measurements need to distinguish the candidate from the working version: a small failing group can disappear inside healthy overall numbers.
Applied to an AI workflow, our recommendation is to begin with a limited, supervised group after offline checks pass. Compare correction rates, incomplete work, and response time by version. Keep the stop conditions active, and expand only when the observed result supports doing so.
If a parallel comparison would duplicate actions, keep the candidate read-only or use recorded inputs instead. Receiving the same request is not permission for two systems to carry it out.
Check that the fallback is real
Before switching, establish whether the previous model will remain available and whether its configuration still works with the current application. A saved model name is not a recovery plan if the endpoint is gone.
We recommend testing the fallback before it is needed. If returning to the old model is impossible, define a manual path or pause the affected automation. Do not silently route confidential work to an unapproved provider as an emergency substitute.
Returning to the previous version also does not undo actions already taken. Keep enough information to find candidate-produced records that need review, without retaining unnecessary sensitive content.
A successful upgrade leaves the business with a tested configuration, an explanation of what improved, and a practical response if something goes wrong. Tell us which workflow you want to improve. We can help determine whether a model change is the useful next step or whether the problem sits elsewhere in the system.