Evaluations API
Send results from your own evaluator or benchmark harness, and read what Provantix can say from them. All routes need an admin credential. See Evaluated quality for what the results mean and what they do not.
POST /v1/evaluations
Send up to 500 results in one request.
POST /v1/evaluations
authorization: Bearer <admin credential>
content-type: application/json
{ "results": [ <result>, ... ] }
A result is a case or a summary. Fields:
| Field | Required | Meaning |
|---|---|---|
external_id |
yes | Your own id for this result: letters, digits and _ . : / -, up to 200 characters. With the evaluator name it is the idempotency key. |
evaluator |
yes | { "name", "version"? }. |
dataset |
yes | { "id", "version" }. Both required. |
rubric |
no | { "id", "version" }. |
agent_id |
yes | The agent evaluated. |
model |
yes | { "id", "version"? }: the model the agent ran on. |
metric |
yes | { "name" }. Version 1 measures pass rates. |
configuration |
no | { "flag", "variant" }: the flag variant the evaluation ran under. Both are required if it is sent. Windows that ran under different configurations are not compared, and a result without one does not match a result with one. See Configuration. |
case_id and passed |
a case | One case, with passed true or false. sample_count and pass_count must not be sent with it. |
sample_count and pass_count |
a summary | A whole number of observations (1 to 1,000,000) and how many passed (0 to sample_count). case_id and passed must not be sent with it. |
score |
no | A number, kept as reported. The comparisons use pass and fail. |
evaluated_at |
yes | When the evaluation ran, as a date and time. Not more than five minutes in the future. |
window |
no | { "start", "end" }, dates and times. |
task_id |
no | Up to 200 characters. |
evidence_ref |
no | A pointer into your evaluator's own system, up to 300 characters. Never content. |
evidence_sha256 |
no | A 64 character hex digest of what the pointer refers to. |
Any other field is refused, in the result or in a nested object, so a prompt, an answer, a dataset row or personal data cannot be sent by accident.
Response. 201 when at least one result was recorded or recognised as a repeat, 400 when none was or the request is not a list of results, 413 over 500. The body lists each item:
{ "schema": "provantix.evaluation.v1", "recorded": 2, "duplicates": 1, "rejected": 1,
"items": [ { "index": 0, "status": "recorded", "result_id": "evr_..." },
{ "index": 1, "status": "duplicate", "result_id": "evr_..." },
{ "index": 2, "status": "rejected", "reason": "..." } ] }
One bad item never stops the others. The same external_id from the same evaluator with the same content is a duplicate; with different content it is rejected, so a retry can never rewrite what you sent.
GET /v1/evaluations
The latest results, newest first. Query: agent_id (optional), limit (default 200, at most 1000).
GET /v1/evaluations/cohorts
For each agent, evaluator, dataset and metric, whether quality fell. Query: agent_id (optional). Each cohort has a status (degraded, improved, stable, insufficient_evidence, stale_source or incomparable), the baseline and current windows with n, passed, rate and a 95% interval, the threshold, a reason where the status is not a plain comparison, and the source statement.
GET /v1/evaluations/compare
The same cases under two model versions. Query, all required: agent_id, evaluator, dataset, metric, from, to (a model as id or id@version).
When the comparison can be made, comparable is true and the response has matched, unmatched, from_rate and to_rate (with intervals), change_points, regressions and fixed (each with count and the cases, every one with its result ids and evidence pointers), hidden_regression, a statement, and reviewer_detection and operational_impact, each { "measured": false, "reason": ... }. Otherwise comparable is false with the reason: a differing dataset version, rubric or evaluator version, only summaries, or fewer than 20 matched cases.
POST /v1/evaluator-calibrations
Report how an evaluator version did on cases with a known answer. Up to 100 per request.
| Field | Required | Meaning |
|---|---|---|
external_id |
yes | Your id for this calibration. With the evaluator name it is the idempotency key. |
evaluator |
yes | { "name", "version" }. Both are required: a calibration belongs to one evaluator version. |
case_set |
yes | { "id", "version" }: your name for the set of known cases. |
calibrated_at |
yes | When the evaluator ran on them. Not in the future. |
should_pass |
yes | { "total", "right" }: of the cases that should pass, how many it passed. |
should_fail |
yes | { "total", "right" }: of the cases that should fail, how many it failed. |
evidence_ref, evidence_sha256 |
no | A pointer into your system and a digest. Never the cases. |
Any other field is refused. The response lists each item as recorded, duplicate or rejected, like the evaluations route.
GET /v1/evaluator-calibrations?evaluator=<name> lists them. GET /v1/evaluator-calibrations/status?evaluator=<name>&version=<version> returns the status (calibrated, weak, insufficient, stale or not_calibrated), the counts with intervals and the reason, from the latest calibration of that exact version.
Disputes
POST /v1/evaluation-disputes with { "result_id": "evr_...", "reason": "..." } disputes one result, or { "subject": "agent:evaluator:dataset:metric", "reason": "..." } disputes a whole series (the subject of its finding). Exactly one of the two. One open dispute per target. The response is 201 with the dispute.
POST /v1/evaluation-disputes/:id/settle with { "outcome": "evaluator_wrong" | "evaluator_right" | "unresolved", "resolution": "..." } records how it was settled, as the sender's statement. POST /v1/evaluation-disputes/:id/withdraw withdraws an open one. GET /v1/evaluation-disputes?status=&result_id=&subject= lists them.
A result with an open dispute, or one settled as evaluator_wrong, is set aside in the second reading of its series (without_disputed in the cohorts) and flagged on the cases of a model comparison. The result itself is never edited.
GET and PUT /v1/evaluation-rules
GET returns the version in force, the rules, who set it and why, and the history (version 0 is the built-in defaults). PUT { "rules": { "drop_points": 8 }, "reason": "..." } changes only the rules named and makes a new version. A reason of at least five characters is required, and a change that alters nothing is refused with 409.
| Rule | Default | Range |
|---|---|---|
recent_days |
14 | 1 to 90 |
baseline_days |
30 | 1 to 365 |
min_observations |
30 | 5 to 1,000 |
drop_points |
5 | 1 to 50 (percentage points) |
min_matched_cases |
20 | 5 to 1,000 |
calibration_floor |
80 | 50 to 99 (percent of known cases right) |
calibration_min_cases |
20 | 5 to 500, of each kind |
calibration_max_age_days |
90 | 7 to 730 |
What a response never contains
Provantix certifies none of it. Every response carries source: "reported by