Recovery after a change: a rollback is not proof
When an agent's behaviour regresses, the fix is usually a configuration change made somewhere else: a feature flag platform rolls a prompt back for the customers who were affected. The change happening is not the same as the agent recovering. A rollback can miss the cohort, the old variant can still be served to some runs, the step that was skipped can still be skipped, and quality can stay low while every infrastructure metric is green.
Provantix does two things here, and only two. It records the change as your control system reports it, and it checks whether the same cohort recovered, from the runs that came after. It never makes the change, schedules it or approves it. Whatever is authorized to change production keeps doing that.
What is recorded
Your control system sends each change to POST /v1/interventions: the agent, the flag, the variant it was on and the variant it now serves, what kind of change it was (rollback, rollout or change), when it happened, who or what made it, and optionally when the affected configuration began serving and which cohort it was for. It is kept as that system's report, never edited, and a repeat of the same report is recognised.
The cohort is a cost tag name and value, for example plan=gold. Leave it out and the change is for every run of the agent. The "who" is an id or a service name, never an email address: Provantix keeps no personal address here.
What is checked
On the agent's page, open Changes. For each reported change Provantix looks at three periods, always for the same agent and the same cohort and never at an average that mixes in other traffic: the runs from before the regression (the reference), the runs during it, and the ended runs since the change.
| Check | What it asks |
|---|---|
| Runs are on the expected variant | Of the runs after the change that report the flag, how many carry the variant the change says they should be on? A run that does not report the flag is counted separately and is never assumed to be served. |
| Declared steps are back | For each expected step declared for the agent, how many of the runs after the change include it, against the share before. |
| Quality is back | Did the outside evaluator's results come back to where they were before the regression, under the same stated configuration and the same evaluation definition? A different dataset, rubric, evaluator version or model is not comparable, never silently joined. These are the evaluator's reported results, not a score Provantix certifies. |
| Nothing new got worse | Errors, run time (95th percentile), tokens and cost per run, against the runs from before the regression. |
| It held | Does the later half of the runs since the change still pass the variant and step checks, on enough runs? |
The verdict
- Recovered: every check passes with enough runs behind it.
- Not recovered: at least one check fails. The page names which.
- Not enough evidence yet: a check has too few runs, no evaluation results, or no flag reports. This is never read as a pass. A rollback event on its own, with nothing after it, is exactly this.
- Not comparable: the only quality figures that exist were scored under a different definition.
The defaults are at least 30 runs per check (15 for the later half, and 30 evaluation observations), 95% of flag-reporting runs on the expected variant, at most 5% of runs missing a declared step, a rise of at least 2 percentage points in errors that the intervals confirm, and a rise of 25% in run time, tokens or cost. The response lists the thresholds it used.
What this does not say
It does not say what caused the regression or the recovery. It does not say the change was the right one, or that the agent's answers are correct. The change is the control system's report; the run figures are recorded telemetry; quality is the evaluator's report. Where a flag was not sent with a run, or the evaluator reported nothing under that configuration, the check says it could not tell.
Where else it shows
- Incidents. A change reported for an agent in an incident, within a week of its records, is on the incident timeline as a configuration change, with its verdict beside it. The timeline still does not claim that anything fixed the incident.
- Follow-ups. A follow-up on a finding shows the changes reported for the finding's agent while it ran, with each verdict. It does not change the follow-up's own outcome, which is about the finding's condition.