Evaluations API

Send results from your own evaluator or benchmark harness, and read what Provantix can say from them. All routes need an admin credential. See Evaluated quality for what the results mean and what they do not.

POST /v1/evaluations

Send up to 500 results in one request.

POST /v1/evaluations
authorization: Bearer <admin credential>
content-type: application/json

{ "results": [ <result>, ... ] }

A result is a case or a summary. Fields:

Field Required Meaning
external_id yes Your own id for this result: letters, digits and _ . : / -, up to 200 characters. With the evaluator name it is the idempotency key.
evaluator yes { "name", "version"? }.
dataset yes { "id", "version" }. Both required.
rubric no { "id", "version" }.
agent_id yes The agent evaluated.
model yes { "id", "version"? }: the model the agent ran on.
metric yes { "name" }. Version 1 measures pass rates.
configuration no { "flag", "variant" }: the flag variant the evaluation ran under. Both are required if it is sent. Windows that ran under different configurations are not compared, and a result without one does not match a result with one. See Configuration.
case_id and passed a case One case, with passed true or false. sample_count and pass_count must not be sent with it.
sample_count and pass_count a summary A whole number of observations (1 to 1,000,000) and how many passed (0 to sample_count). case_id and passed must not be sent with it.
score no A number, kept as reported. The comparisons use pass and fail.
evaluated_at yes When the evaluation ran, as a date and time. Not more than five minutes in the future.
window no { "start", "end" }, dates and times.
task_id no Up to 200 characters.
evidence_ref no A pointer into your evaluator's own system, up to 300 characters. Never content.
evidence_sha256 no A 64 character hex digest of what the pointer refers to.

Any other field is refused, in the result or in a nested object, so a prompt, an answer, a dataset row or personal data cannot be sent by accident.

Response. 201 when at least one result was recorded or recognised as a repeat, 400 when none was or the request is not a list of results, 413 over 500. The body lists each item:

{ "schema": "provantix.evaluation.v1", "recorded": 2, "duplicates": 1, "rejected": 1,
  "items": [ { "index": 0, "status": "recorded", "result_id": "evr_..." },
             { "index": 1, "status": "duplicate", "result_id": "evr_..." },
             { "index": 2, "status": "rejected", "reason": "..." } ] }

One bad item never stops the others. The same external_id from the same evaluator with the same content is a duplicate; with different content it is rejected, so a retry can never rewrite what you sent.

GET /v1/evaluations

The latest results, newest first. Query: agent_id (optional), limit (default 200, at most 1000).

GET /v1/evaluations/cohorts

For each agent, evaluator, dataset and metric, whether quality fell. Query: agent_id (optional). Each cohort has a status (degraded, improved, stable, insufficient_evidence, stale_source or incomparable), the baseline and current windows with n, passed, rate and a 95% interval, the threshold, a reason where the status is not a plain comparison, and the source statement.

GET /v1/evaluations/compare

The same cases under two model versions. Query, all required: agent_id, evaluator, dataset, metric, from, to (a model as id or id@version).

When the comparison can be made, comparable is true and the response has matched, unmatched, from_rate and to_rate (with intervals), change_points, regressions and fixed (each with count and the cases, every one with its result ids and evidence pointers), hidden_regression, a statement, and reviewer_detection and operational_impact, each { "measured": false, "reason": ... }. Otherwise comparable is false with the reason: a differing dataset version, rubric or evaluator version, only summaries, or fewer than 20 matched cases.

POST /v1/evaluator-calibrations

Report how an evaluator version did on cases with a known answer. Up to 100 per request.

Field Required Meaning
external_id yes Your id for this calibration. With the evaluator name it is the idempotency key.
evaluator yes { "name", "version" }. Both are required: a calibration belongs to one evaluator version.
case_set yes { "id", "version" }: your name for the set of known cases.
calibrated_at yes When the evaluator ran on them. Not in the future.
should_pass yes { "total", "right" }: of the cases that should pass, how many it passed.
should_fail yes { "total", "right" }: of the cases that should fail, how many it failed.
evidence_ref, evidence_sha256 no A pointer into your system and a digest. Never the cases.

Any other field is refused. The response lists each item as recorded, duplicate or rejected, like the evaluations route.

GET /v1/evaluator-calibrations?evaluator=<name> lists them. GET /v1/evaluator-calibrations/status?evaluator=<name>&version=<version> returns the status (calibrated, weak, insufficient, stale or not_calibrated), the counts with intervals and the reason, from the latest calibration of that exact version.

Disputes

POST /v1/evaluation-disputes with { "result_id": "evr_...", "reason": "..." } disputes one result, or { "subject": "agent:evaluator:dataset:metric", "reason": "..." } disputes a whole series (the subject of its finding). Exactly one of the two. One open dispute per target. The response is 201 with the dispute.

POST /v1/evaluation-disputes/:id/settle with { "outcome": "evaluator_wrong" | "evaluator_right" | "unresolved", "resolution": "..." } records how it was settled, as the sender's statement. POST /v1/evaluation-disputes/:id/withdraw withdraws an open one. GET /v1/evaluation-disputes?status=&result_id=&subject= lists them.

A result with an open dispute, or one settled as evaluator_wrong, is set aside in the second reading of its series (without_disputed in the cohorts) and flagged on the cases of a model comparison. The result itself is never edited.

GET and PUT /v1/evaluation-rules

GET returns the version in force, the rules, who set it and why, and the history (version 0 is the built-in defaults). PUT { "rules": { "drop_points": 8 }, "reason": "..." } changes only the rules named and makes a new version. A reason of at least five characters is required, and a change that alters nothing is refused with 409.

Rule Default Range
recent_days 14 1 to 90
baseline_days 30 1 to 365
min_observations 30 5 to 1,000
drop_points 5 1 to 50 (percentage points)
min_matched_cases 20 5 to 1,000
calibration_floor 80 50 to 99 (percent of known cases right)
calibration_min_cases 20 5 to 500, of each kind
calibration_max_age_days 90 7 to 730

What a response never contains

Provantix certifies none of it. Every response carries source: "reported by , not certified by Provantix".