Evaluated quality fell

What you're seeing

A finding like:

harness-a reports a lower task_pass_rate for agt_refunds on refund-cases: 71% (142 of 200, 64.5% to 76.7%) in the last 14 days, against 89% (178 of 200, 83.8% to 92.8%) in the 30 days before.

It is an Evaluated quality fell finding, medium severity.

What it means

Your own evaluator reported results for one agent, on one dataset, under one model version and one rubric. The recent pass rate is at least 5 points lower than the one before, there are at least 30 observations on each side, and the two 95% ranges do not overlap. Provantix does not run the evaluation or judge the answers. It reports that your evaluator's numbers moved, and that the model, dataset version, rubric and evaluator version are the same in both windows.

What it does not mean: that the agent got worse. The agent, its prompt or tools, the data it sees, or the evaluator itself may have changed.

What to do

  1. Open your evaluator's own results for the cases that now fail. The finding carries no case content on purpose.
  2. Find what changed around the date the figures moved: a prompt, a tool, the data, a deployment, the evaluator.
  3. If the model was upgraded, compare the two versions case by case (the Quality tab on the agent, or GET /v1/evaluations/compare). A better overall figure can hide cases that passed before and fail now; the comparison lists them.
  4. Decide. Roll back, adjust, or accept it. Provantix changes nothing and blocks nothing.
  5. Resolve the finding with the cause if you want it on the record. It closes by itself when the figures recover.

If you see something else instead

The agent's Quality tab says what it is not showing a finding for:

  • Not enough evidence: fewer than 30 observations in a window, or no earlier window yet.
  • Source quiet: your evaluator has reported nothing for this in 14 days.
  • Not comparable: the model, dataset version, rubric or evaluator version differs between the windows. That starts a new series.

See Evaluated quality and the Evaluations API.