Evaluated quality: what your evaluator reports, and what changed after an upgrade

Failure rates and latency tell you whether an agent is running. They do not tell you whether its answers are as good as they were. If you run your own evaluator or benchmark harness, you can send its results to Provantix. They are kept as your evaluator's report: attributed to it and to whoever sent them, with the model version, the dataset and the rubric they were scored against. Provantix does not run evaluations, does not judge answers, and does not certify that a score is right.

In the product

Agent Quality tab before evaluation results have been sent

Product capture, 15–16 November 2026. Open the image for full size. Counts and dates are examples from the captured workspace.

Open an agent and select Quality. This example has no evaluation results, so no score is shown. The comparison fields select evaluator, dataset, metric and model versions; they do not supply evaluation evidence themselves.

What you send

One request, up to 500 results, with an admin credential:

POST /v1/evaluations
{ "results": [
  { "external_id": "run-7731-case-0042",          your own id: makes a repeat safe
    "evaluator": { "name": "harness-a", "version": "3.1" },
    "dataset":   { "id": "refund-cases", "version": "2026-10" },
    "rubric":    { "id": "refund-rubric", "version": "4" },       optional
    "agent_id": "agt_...",
    "model":     { "id": "sre-llm", "version": "2.0" },
    "metric":    { "name": "task_pass_rate" },
    "case_id": "case-0042", "passed": false,                      one case, pass or fail
    "evaluated_at": "2026-11-12T09:30:00Z",
    "evidence_ref": "harness-a://runs/7731" } ] }

A result is either one case (a case_id and passed) or a summary (sample_count and pass_count, no case id). The first version measures pass rates. Provantix stores ids, versions, a pass or fail and a pointer into your evaluator's own system. Any other field is refused, so a prompt, an answer, a dataset row or personal data cannot be sent by accident. An optional score is kept as reported and not used in the comparisons.

The same external_id from the same evaluator is recognised as a repeat and not stored twice. The same id with different content is refused. Results are append-only. Both the time the evaluation ran and the time Provantix received it are kept, and a late result is placed by when it ran.

Has quality fallen? (a finding)

For one agent, one evaluator, one dataset and one metric, Provantix compares the last 14 days with the 30 days before. A finding, Evaluated quality fell, is raised only when:

  • there are at least 30 observations in each window,
  • the pass rate fell by at least 5 points, and
  • the 95% ranges of the two windows do not overlap.

The finding shows the baseline, the current figure, both denominators, the ranges, the threshold and the source. Otherwise the page says what is true instead:

Status Meaning
no clear change The difference is within what the numbers can tell from noise, or under the threshold.
rose The mirror of a fall. Not a finding.
not enough evidence Too few observations, or no earlier window yet.
source quiet The evaluator has reported nothing in the last 14 days, so there is no current figure.
not comparable The model, the dataset version, the rubric or the evaluator version differs between the windows. That starts a new series and is never joined silently.

Two evaluators that disagree are shown as two separate rows. They are never averaged.

What changed when the model changed? (a comparison)

A model upgrade can raise the overall pass rate while breaking cases that used to pass. To see it, compare two model versions on the same cases:

GET /v1/evaluations/compare?agent_id=...&evaluator=...&dataset=...&metric=...&[email protected]&[email protected]

You get the cases scored under both versions, the pass rate under each with its range, the cases that passed before and fail now, listed one by one with a pointer to the evaluator's record, and the cases that went the other way beside them. If the overall figure did not fall but some cases now fail, it says so. The Quality tab on an agent shows the same.

A comparison is refused, with the reason, when the dataset version, the rubric or the evaluator version differ between the two sides, when only summaries exist (they cannot be matched case by case), or when fewer than 20 cases were scored under both.

Is the evaluation itself reliable?

A result you can verify is not a result that is right. Provantix keeps three more things about the evaluation, so a conclusion can be read together with what it rests on. It runs no evaluator, chooses no cases and decides no dispute.

Calibration. The evaluator runs a set of cases whose answer is known, some it should pass and some it should fail, and reports how many it got right (Evaluations API). It is kept per evaluator version: the next version is not calibrated by the last one. A finding then says whether the evaluator version is calibrated (passed and failed enough known cases), weak (below the floor in the rules), insufficient (too few known cases), stale (older than the rules allow) or not calibrated. These are the evaluator's reports; Provantix has not checked the cases.

Disputes. A person can dispute one result or a whole series, with a reason. The result is never edited. A series with disputed results is read a second time with those results set aside, and the finding says what that reading shows, for example that the fall disappears without them. The evaluator's owner settles the dispute (the evaluator was wrong, it was right, or it is not resolved), and that is recorded as their statement. A result settled as wrong stays set aside; one settled as right counts again.

Rules. The thresholds a finding is judged by (the windows, the observations needed, the fall that counts, the cases needed to compare versions, and the calibration floor, size and age) are an organization setting. Each change is a new version with who made it and why, and a history. A finding names the rules version it came from, and a follow-up is not comparable across versions: a finding that stopped because the rules changed is not reported as improved.

What is not measured

Two things a model change can affect are not measured, and the comparison says "not measured", not zero:

  • Whether reviewers noticed the new errors. That needs a record of what the people reviewing the agent's work caught. Provantix has none from your evaluator.
  • Whether outcomes in the workflow changed, such as reversals, escalations or failures, tied to this upgrade.

What this does not do

  • It does not detect hallucination. It reports that your evaluator's pass rate changed. Whether the evaluator is right is outside what Provantix can show.
  • It does not run your evaluator, choose its known cases or decide a dispute. It records the evaluator's calibration, a person's dispute and the owner's settlement, and shows them beside the finding.
  • It does not block or approve an upgrade. You keep that decision. Provantix shows the evidence.
  • It does not duplicate failure or latency monitoring. Those stay in findings, in one place.
  • It is not yet in signed exports or the offline verifiers. The results are stored and shown, but an export does not carry them yet.