Metrics definitions
Every number in the product is computed from sealed records and carries its coverage denominator (see Data expectations and the Settings → Data coverage card). This page defines each metric, what it includes, and what it excludes.
Model calls and tool calls are separate
Metrics are reported per record kind, never blended:
- Model calls — records with
action_type = call_model. Tokens, latency and failure rate are computed over these only. - Tool calls — records with a tool name or a
call_tool/sendaction. Target, recipient and summaries apply here. - Other actions — everything else (logins, grants, admin operations). Only outcome coverage applies.
A tool call that internally invokes a model counts as a tool call for tool metrics; the model invocation only appears in model metrics if the instrumentation emits it as its own call_model record.
Failure rate
failures / records, per day, over model calls unless stated otherwise. A failure is a record whose outcome is failure — the call raised or returned an error.
What it excludes: task failure. success means "no error recorded", not "task completed". An agent that approves the wrong purchase order cleanly is a success. See the quality field.
p95 latency
The 95th percentile of latency_ms over model calls that carry a latency value, per day. Tool-call latency is reported separately where present.
What it excludes: calls without a reported latency — and the coverage card says what fraction that is. A p95 drawn over 40% of calls describes that 40%.
Tokens
tokens_in and tokens_out sums over model calls that report them, per day. Token counts come from the emitter's own usage reporting; Provantix does not recount them against provider billing.
Cost (estimate)
Cost is estimated at ingest, per record:
plain input / 1000 × input_per_1k
+ cache read / 1000 × cached_input_per_1k
+ cache write / 1000 × cache_write_per_1k
+ output / 1000 × output_per_1k (reasoning tokens at reasoning_per_1k when that is set, else as output)
× the batch, priority or flex multiplier, when the call used that tier
where plain input is the input tokens that were not read from or written to a cache. A call that reported no cache, reasoning or tier prices as input and output only, exactly as before. See Token kinds and how each is priced.
- Usage source: the token counts the emitter reported with the record.
- Pricing source: your organization's price table (add rows under Settings, Cost tab, Model prices, or with
POST /v1/prices). Each row has a provider, model, price per 1k input/output tokens, optional cached input, cache write and reasoning prices and batch, priority and flex multipliers, and an effective date. - Calculation version: the pricing function itself is versioned with the schema; a record is priced at insert, and its estimate is recalculated when a price row changes, so a past estimate can change. The stored pricing reference names the price row in force, not the rates it had.
- Currency: USD.
- Reconciliation: nothing reconciles these figures against the provider's invoice. Cost is directional — good for chargeback splits, budget alerts and trends — not a billing record. When billing reconciliation ships, it will appear as a separate, labelled comparison, not as a silent rewrite of history.
Unpriced models and records without token counts show no cost (NULL), never zero — a zero would claim a priced call that was free. A call that used a cache, reasoning or a tier with no price set for it is priced for the parts that have one, and is counted as unpriced: its cost is at least what is shown.
Coverage percentages
Every chart that shows a metric over a subset of records states its denominator: "latency known for X% of N model calls". The fraction is computed over the same window as the chart. A low percentage is a telemetry gap (see Data expectations), not a small sample of an otherwise complete picture.
Cost by workflow and by requester
A workflow is one trace: every record sharing a trace id. Its cost is the sum of the priced model calls in it, and its unpriced calls are counted beside the cost, so the figure is a floor when any are unpriced and has no value, not zero, when all are. A requester is the first human handle in a record's on-behalf-of list. A record with none, or one that was erased, is counted as "No requester recorded". Both are grouped from the same per-record estimate as every other cost figure (see above) and are directional in the same way.
Step timing in a run
A step's start is its observed time, which for a traced call is when the call started. Its duration is the source's own: end minus start for an OpenTelemetry span, and absent for a source that does not send one. The run length on the timeline is the latest end of any step. An idle gap is the time between the end of the step that has finished latest so far and the start of the next step, and it is measured only when that step recorded a duration.
Same call repeated
Five or more consecutive records of one agent, in one trace (or, with no trace, one session), with the same action type, tool and target, each no more than five minutes after the one before, and nothing different in between. Model calls, and calls with neither a tool name nor a target, are not counted. The last 24 hours are examined. An agent whose owner declared how often it runs is not reported for a run whose middle gap is a minute or more.
Evaluated quality
For one agent, one evaluator, one dataset and one metric: the pass rate is passes divided by observations (a case result is one observation; a summary result counts its sample_count). The recent window is the last 14 days and the baseline is the 30 days before it. Each rate has a 95% Wilson score interval, which is wider for fewer observations. The quality finding is raised only when each window has at least 30 observations, the pass rate fell by at least 5 points, and the two intervals do not overlap. A model, dataset version, rubric or evaluator version that differs between the windows makes the comparison "not comparable" and starts a new series.
For a model version comparison, cases are matched by case_id, the latest result of each case under each version stands, and at least 20 cases must be scored under both. A regression is a case with pass_count 1 under the old version and 0 under the new; a fix is the reverse. Reviewer detection and operational impact are reported as not measured because the product has no source for them.
Calls failing in a row
In one trace (or, with no trace, one session), an agent's most recent calls that ended in failure, three or more in a row, with the first no more than 30 minutes ago. Calls with no recorded outcome are not counted either way, and a later success ends the run. Model calls count. When the same call was repeated five or more times the run is reported as a repeated call and not again here.
Fleet states
Decided in this order from each agent's records: its lifecycle if it is not running (proposed, paused, retired); waiting when its latest enforcement verdict in the last 24 hours is ask and it has recorded nothing since; failing when its most recent three or more outcome-bearing calls failed, the latest within 30 minutes; working when it recorded something in the last 5 minutes; idle when it did in the last 24 hours; quiet otherwise. Verdict and decision records are not counted as the agent's activity.
Spend rate
The estimated cost and output tokens of model calls over the last 5 minutes and the last 60 minutes, divided by the window (cost per minute, tokens per second), and a series by minute. The current minute is included as a whole minute. Priced calls are summed; model calls with no price are counted beside the rate and never estimated.
Budget levels and roll-ups
A budget is a monthly cap, a daily cap or both on one level: agent (that agent's records), owner (every agent with that owner), team (every agent carrying that team label) or organization (every recorded cost). Month to date runs from the first of the UTC month, and today is the UTC day. A cap is crossed when spend is strictly above it. The roll-up adds the agents of a team to give the team, and the teams do not necessarily add up to the organization, which also includes records attributed to no agent. All of it is recorded cost, an estimate, and a budget blocks nothing.
Expected step missing
For one declaration (an agent, a tool, optionally a system): the runs of that agent (one trace each) that started at or after the declaration and within the last 7 days. A run is ended when its latest record is more than 30 minutes old, and only ended runs are counted. A run is included when it has a record of the agent in that trace with the tool name (and the system, when one is named), whatever its outcome. The finding reports the ended runs not included, the ended runs included, and the runs in progress. Decision and verdict records do not make a run. A run that began more than a day before the declaration is not judged.