How to Update an AI Feature in Production Without Breaking It
Treat changes to prompts, models, retrieval, and tool permissions as releases: record what changed, test it against the same representative cases, roll it out with a stop rule, and keep a tested path back.
Key Takeaways
- Treat each change that can alter an answer or action as a release, including prompts, model identifiers, retrieval data, tool schemas, permissions, and orchestration settings.
- Keep a release record that connects the exact configuration to its evaluation results, rollout owner, stop conditions, and rollback target.
- Compare a candidate with the current version on representative cases, including failures and access boundaries; do not rely on a single aggregate score.
- Use shadow or staged rollout only when you can protect data, prevent duplicate side effects, and gather enough useful signal for the feature's risk and volume.
- A rollback is only useful if it restores the dependent components together and the team has practiced the procedure.
Update a production AI feature the way you release any high-impact software change: identify every component that can affect behavior, evaluate the candidate against a known baseline, stage exposure when practical, and keep a tested way to restore the last acceptable version. A prompt edit, provider model change, retrieval refresh, or tool-permission adjustment can alter answers even when the application still returns HTTP 200.
The useful unit of release is therefore not just the code commit. It is the set of code, model, prompt, data, and access-control versions that produced the behavior you are shipping. This guide gives product and engineering teams a lightweight release record and a decision process for changing that set safely.
What can change the behavior of a live AI feature?
Start by listing the moving parts in the feature. The answer may include the model and request settings, system and task prompts, retrieved documents, chunking or ranking configuration, tool definitions, authorization checks, orchestration logic, and fallback behavior. A release that changes one of these can affect quality, cost, latency, or the actions the feature is allowed to take.
| Change | What to compare | Release check |
|---|---|---|
| Prompt or policy | Task completion, required format, refusals, and known edge cases | Run the affected evaluation cases and inspect important regressions |
| Model or request settings | Quality, latency, cost, tool behavior, and provider constraints | Test the exact provider, identifier, and settings planned for release |
| Retrieved data or index | Relevant results, coverage, freshness, and tenant boundaries | Test representative queries and verify access rules before switching |
| Tools or permissions | Valid arguments, allowed actions, denied actions, and approval paths | Test both permitted and prohibited operations using realistic identities |
A change can span more than one row. For example, a new model may respond differently to an existing tool schema. Treat coupled changes as one candidate release so the evaluation tests the combination customers will actually use.
Create a release record before changing production
Keep a small record for each candidate. It should make the change reviewable by someone who did not author it and answer four questions: what changed, what evidence supports release, how will the team detect harm, and what exactly will it restore?
Do not put raw customer prompts, personal information, or confidential outputs into a general release note just to make it detailed. Store evaluation data and traces under the product's access, retention, and redaction rules; the release record can point to controlled systems instead.
A hypothetical example: refreshing a support assistant
Suppose a B2B software team wants to update the instructions and help-center index used by its support assistant. The release record identifies the prompt revision, source-document snapshot, index build, application commit, and current model identifier. The evaluation set includes ordinary questions, a recently changed policy, a question the docs cannot answer, a request to expose another customer's information, and a request that should be handed to a person. The team compares the candidate with the live version, confirms the answers cite the intended documents, then enables the candidate for an internal support group before expanding access. If citations become unreliable or an access check fails, the team switches back to the saved prompt and index configuration. This example is illustrative; it does not describe a Game Changer Labs client deployment.
Compare the candidate with a fixed evaluation baseline
Run the proposed change and the current production configuration on the same representative cases. Include common successful inputs, ambiguous requests, known failure examples, boundary conditions, and security or permission checks. If the feature uses tools, verify both the decision to call a tool and the validity and authorization of its arguments. If it retrieves customer data, check that useful evidence is found and that one tenant cannot access another tenant's material.
Define pass conditions around the feature's job. A support-draft assistant might be judged on whether it cites the right policy, follows account permissions, and escalates uncertain cases; a classification feature may need precision and recall checks by important category. There is no single score or test count that fits every feature. Use human review for judgments that an automated grader cannot reliably make, and inspect the cases where the candidate differs from the baseline rather than relying only on an average score.
If the candidate fails a critical authorization or safety check, stop. A higher average quality score does not cancel out a severe boundary failure. For more detail on building the test set and reviewing results, see our guide to evaluating and testing AI agents.
Choose offline comparison, shadow mode, or staged exposure
The safest release method depends on the impact of an error, how much traffic the feature receives, and whether the candidate can be evaluated without exposing customers. Use an offline dataset when it is representative and permitted. Consider shadow mode when production inputs reveal cases your dataset misses, but remember that shadow mode still processes data and consumes resources. The candidate must not send emails, modify records, or trigger other actions while it is being observed.
A staged rollout can reduce the number of users affected by a faulty change. Pick rollout steps based on the product's architecture, customer segments, and the amount of evidence needed. Google Cloud's canary deployment guidance describes routing a portion of traffic to a new version and increasing exposure in stages; that is a useful pattern, not a universal percentage prescription. If a feature has little traffic, a small canary may produce too few observations to be informative. In that case, use deeper offline checks or a deliberate pilot with users who understand the change.
Agree on the signals before release. Depending on the feature, watch task completion or correction rates, safety and permission incidents, provider errors, latency, cost per task, support feedback, and fallback use. Choose a comparison window that reflects how often the workflow occurs; a brief quiet period is not evidence that a low-frequency feature is healthy.
Keep the old path available while the new one is being evaluated, where the architecture and data lifecycle allow it. Staged rollouts work only when traffic can be routed cleanly and both paths remain operational. Do not create a second path that silently bypasses authorization, audit, or retention controls.
Track model and retrieval lifecycle changes explicitly
Record the provider, model identifier, API region or endpoint where relevant, and the request settings used by the feature. Providers publish model catalogs and lifecycle notices; for example, Anthropic documents deprecations and retirement dates for Claude API models, while OpenAI documents its available API models. Check the current provider documentation when planning a change. A specific identifier helps make an upgrade deliberate, but it does not guarantee identical outputs across runs or freeze every service dependency.
For retrieval-augmented features, version the data and the retrieval configuration alongside the model: source snapshot or ingestion run, chunking, embedding model, filters, ranking settings, and index. Test that the new configuration retrieves the expected evidence and preserves access boundaries. If you change embedding dimensions or representation, plan how existing vectors will be rebuilt or separated; don't mix incompatible vectors in the same query path. Keep the previous index or a recoverable source snapshot until the new path has passed its checks.
Treat a model or corpus upgrade as a distinct release even if application code barely changes. Teams that are changing providers as well can use the separate AI model provider migration guide for the migration-specific sequence.
Write and practice a rollback plan
Set the stop conditions before the rollout starts. Examples include a critical permission failure, harmful output in a high-impact workflow, a material drop in a required task measure, an unexpected cost increase, or a provider error that breaks the user journey. Define thresholds against the product's own baseline and risk tolerance; generic numbers from another team may create false alarms or miss the failures that matter here.
Specify what to restore together. Reverting only the prompt may not help if the new model requires a different tool schema; switching back to a previous embedding model may not work against a newly built index. Document how to return the model, prompt, retrieval pointer, permissions, and feature flag to a compatible state. Also decide what happens to in-flight requests and any action that may already have occurred.
Finally, rehearse the operation in the environment where releases actually happen. Confirm who can pause traffic, who can restore configuration, what alerts they can see, and how the change is recorded after the incident. A runbook is useful only if an on-call teammate can follow it with the access they have.
A practical release sequence for a B2B product team
- Describe the user-visible change and list every behavior-affecting component.
- Save the current configuration and baseline evaluation results as the rollback reference.
- Run the candidate against representative normal, boundary, failure, and authorization cases.
- Review material differences, including quality, cost, latency, refusals, retrieval, and tool actions.
- Choose offline, shadow, staged, or direct release based on risk and available evidence.
- Watch the agreed signals, stop when a release condition is met, and restore the compatible prior configuration if needed.
- Turn confirmed production failures into new evaluation cases and update the release record.
This sequence is a general framework, not a claim about a particular customer deployment. The right controls depend on the product, its users, and the impact of an incorrect answer or action. For the wider production-readiness picture, see AI agent observability and scaling an AI pilot.
If your team has a live AI capability and needs help making changes testable and reversible, describe the feature and the change you are planning. Include the product workflow, the systems or data involved, and what a safe release would need to prove.
Frequently Asked Questions
What should be versioned when updating an AI feature?
Record every component that can change the feature's behavior: application and orchestration code, prompt or policy configuration, model provider and model identifier, tool schemas, retrieval corpus and index version, embedding configuration, permission rules, and relevant feature flags. Link those versions to the evaluation results and release decision so the team can identify what produced an output and restore a compatible configuration.
Do I need to run evaluations before every prompt change?
Run the cases relevant to the change before release. A small wording adjustment may need a focused set of format and task checks; a change to permissions, tool use, or model behavior needs broader coverage. Keep a fixed baseline for comparison, include known failures and boundary cases, and use human review where automated scoring cannot judge correctness or safety reliably.
When should I use shadow mode for an AI release?
Shadow mode can help compare a candidate against real input patterns while continuing to serve the current result. Use it only when your data agreements and privacy controls allow the extra processing, candidate outputs cannot trigger tools or other side effects, and the additional inference cost is acceptable. For sensitive inputs, a representative offline dataset may be the safer comparison.
How much traffic should go to an AI canary?
There is no universally useful percentage. Choose a cohort and rollout step that limit potential impact while producing enough relevant observations to assess the feature. If traffic is low, segment sizes are small, or failures are rare but severe, rely more on offline evaluations and a controlled user pilot; a tiny traffic slice can look reassuring without testing the cases that matter.
Does pinning a model identifier make AI outputs reproducible?
No. A provider's specific model identifier can make the selected model version more explicit and help you control upgrades, but it does not make a probabilistic service deterministic or freeze every provider-side dependency. Record the identifier and request settings, monitor provider lifecycle notices, and evaluate any replacement before switching. Check the provider's current documentation because identifiers and availability change.
What should an AI rollback restore?
Restore a known compatible set: application or orchestration code, prompt and policy configuration, model identifier and request settings, tool contracts, retrieval index or corpus pointer, and relevant permissions or flags. Decide what to do with requests already in flight, queued work, and user-visible outputs. Test the rollback in the real deployment path and document who can trigger it.
Free Tools
Tell us what you're building — book a free scoping call.
Pick a time that works and walk us through your project — 30 minutes, straight to the point. You leave with a concrete plan, timeline, and cost. No sales pitch — if we're not the right fit, we'll say so.
Keep Reading
Get new playbooks by email
Occasional, no-fluff field notes on building production AI — new guides and tools, straight to your inbox. Unsubscribe anytime.