Contracts, Receipts, Evidence, Statistics and Export

Contracts

See Contracts v2.

toolscore.contract.evaluate_contract(contract, actual, weights=None, strict=False, final_answer=None, honesty=None)[source]

Evaluate tool calls against a contract.

Parameters:

  • contract (Contract | dict[str, Any]) – A Contract or a contract mapping.

  • actual (list[dict[str, Any]] | Any) – The calls (dicts, or any raw format toolscore.evaluate() reads).

  • weights (dict[str, float] | None) – Score weights (default: the contract’s).

  • strict (bool) – Exact argument comparison.

  • final_answer (str | None) – The agent’s final answer, for the honesty checks.

  • honesty (HonestyConfig | None) – Honesty-check configuration.

Return type:

EvaluationResult

Returns:

The evaluation of the best combination of alternatives, with metrics["contract_metrics"]: contract_version, alternatives (the option chosen per group and every combination’s score), optional_matched (the optional calls left out of the score) and the counts of required, optional and forbidden entries. trace_calls holds every call, optional ones included.

toolscore.contract.load_contract(path)[source]

Read a contract file (JSON, or YAML with tool-scorer[yaml]).

Return type:

Contract

Parameters:

path (str | Path)

toolscore.contract.contract_from_mapping(data)[source]

Validate and build a Contract from parsed JSON (or a Python dict).

Raises:

ValueError – On a missing or unknown key, or a malformed call.

Return type:

Contract

Parameters:

data (Any)

toolscore.contract.is_contract(data)[source]

Whether parsed JSON is a contract (it has contract_version, or required/optional).

Return type:

TypeGuard[dict[str, Any]]

Parameters:

data (Any)

toolscore.contract.matcher_from_json(value)[source]

Turn $ operators in a JSON value into matchers (see the module docstring).

Return type:

Any

Parameters:

value (Any)

toolscore.contract.split_optional(actual, required, optional)[source]

Leave out optional calls that are not needed to fill a required call.

Walks the calls in order. A call matching an optional entry is dropped when the required calls of its tool are already filled by earlier calls (or the tool is not required at all); otherwise it is kept and fills one.

Return type:

tuple[list[dict[str, Any]], list[int]]

Returns:

(scored calls, indexes of the optional calls left out).

Parameters:
class toolscore.contract.Contract(required=list(), optional=list(), forbidden=list(), weights=None, min_score=0.9)[source]

A parsed contract (see the module docstring).

Variables:
Parameters:
required: list[dict[str, Any] | AlternativeGroup]
optional: list[dict[str, Any]]
forbidden: list[dict[str, Any]]
weights: dict[str, float] | None = None
min_score: float = 0.9
property groups: list[AlternativeGroup]

The alternative groups, in order.

expansions()[source]

Every combination of alternatives: (required calls, {group id: option index}).

In declaration order (the first option of every group first).

Raises:

ValueError – With more than MAX_COMBINATIONS combinations.

Return type:

list[tuple[list[dict[str, Any]], dict[str, int]]]

__init__(required=<factory>, optional=<factory>, forbidden=<factory>, weights=None, min_score=0.9)
Parameters:
Return type:

None

class toolscore.contract.AlternativeGroup(id, options)[source]

A place in the required calls where any one of several call lists is acceptable.

Parameters:
id: str
options: list[list[dict[str, Any]]]
__init__(id, options)
Parameters:
Return type:

None

Receipts and regrade

See Receipts & Regrade.

toolscore.receipts.build_receipt(result, inputs=None, settings=None, final_answer=None)[source]

Build the receipt of one evaluation.

Parameters:

  • result (EvaluationResult) – The evaluation.

  • inputs (dict[str, str | Path | None] | None) – Input files by role ("gold", "trace", "forbidden", "honesty_config", "injected"…); None entries are left out.

  • settings (dict[str, Any] | None) – Settings that changed the result (format, strict…). The normalized weights are always added.

  • final_answer (str | None) – A final answer given from outside the trace; only its hash is stored.

Return type:

dict[str, Any]

Returns:

The receipt dict, with receipt_sha256 last.

toolscore.receipts.write_receipt(path, receipt)[source]

Write a receipt as indented JSON and return its path.

Return type:

Path

Parameters:
toolscore.receipts.load_receipt(path)[source]

Read a receipt file.

Raises:

ValueError – If the file is not a receipt.

Return type:

dict[str, Any]

Parameters:

path (str | Path)

toolscore.receipts.verify_receipt(receipt, base=None)[source]

Check a receipt against itself and the files it names.

Parameters:

  • receipt (dict[str, Any]) – The receipt.

  • base (str | Path | None) – Directory that relative input paths are resolved against (the receipt’s own directory, usually); the current directory if None.

Return type:

list[str]

Returns:

Problems found, empty when the receipt and every input check out. A missing input is a problem; so is one whose bytes changed.

toolscore.receipts.regrade(source, gold=None, weights=None, forbidden=None, format=None, honesty=None, final_answer=None, verify=True)[source]

Re-score a saved trace with a new gold file or contract, weights or rules.

Nothing is re-run. source is a receipt (its trace, gold, format and weights are the defaults) or a trace file (then gold is required).

Parameters:

  • source (str | Path) – A receipt file or a trace file.

  • gold (str | Path | None) – The new gold file or contract (default: the receipt’s).

  • weights (dict[str, float] | None) – New score weights (default: the receipt’s).

  • forbidden (str | Path | None) – A forbidden-rules file (default: the receipt’s, if any).

  • format (str | None) – The trace format (default: the receipt’s, else "auto").

  • honesty (HonestyConfig | None) – Honesty-check configuration.

  • final_answer (str | None) – The final answer, when the trace does not hold one.

  • verify (bool) – Refuse a receipt whose inputs changed (raises ReceiptMismatchError); False regrades anyway and reports the problems.

Return type:

Regrade

Returns:

The Regrade.

Raises:
class toolscore.receipts.Regrade(source, result, receipt, previous=None, problems=None)[source]

One regraded trace.

Variables:

  • source (str) – The receipt or trace that was regraded.

  • result (EvaluationResult) – The new evaluation.

  • receipt (dict[str, Any]) – The new receipt.

  • previous (dict[str, Any] | None) – The previous result block (score, grade…) from the source receipt, or None when regrading a bare trace.

  • problems – What verify_receipt() found (empty when it checked out or verification was off).

Parameters:
source: str
result: EvaluationResult
receipt: dict[str, Any]
previous: dict[str, Any] | None = None
problems: list[str] | None = None
property score_delta: float | None

New score minus the previous one, or None without a previous score.

__init__(source, result, receipt, previous=None, problems=None)
Parameters:
Return type:

None

exception toolscore.receipts.ReceiptMismatchError[source]

A receipt’s inputs changed since it was written, so a regrade would not be faithful.

toolscore.receipts.canonical_json(value)[source]

value as canonical JSON bytes: sorted keys, no whitespace, UTF-8.

Non-JSON leaves (matchers, arbitrary objects) are written as their repr.

Return type:

bytes

Parameters:

value (Any)

toolscore.receipts.json_sha256(value)[source]

The SHA-256 of value as canonical JSON (see canonical_json()).

Return type:

str

Parameters:

value (Any)

toolscore.receipts.file_digest(path)[source]

{"path", "sha256", "bytes"} for a file on disk.

Parameters:

path (str | Path) – The file.

Return type:

dict[str, Any]

Returns:

The digest (path as given, sha256 of the bytes, size in bytes).

Evidence

Tri-state evidence: every check is pass, fail or unknown.

A check that lacks the data it needs does not pass. A trace that records only the requests cannot show that a call succeeded; a trace without a final answer cannot show that the agent disclosed a failure; a model without a known price has no cost. Those checks are unknown, and evaluation_evidence() counts them separately, so a report can say “4 pass, 0 fail, 3 unknown” instead of a clean bill of health.

Checks of an evaluation (evaluation_evidence()):

Check

Pass when

Unknown when

required-calls

every required call was made and did not fail

no calls were required; the calls’ outcomes were not captured

call-errors

no call failed

no call has a result or error (and none failed)

forbidden-calls

no call matched a forbidden rule (only with rules)

never

credentials

no credential in tool arguments

never

claimed-success

no success claim over a failure

no final answer

disclosure

every warning and unrecovered error was disclosed (and every injected fault)

no final answer, or no results captured

escape-hatch

no code-execution call got around a refusal

no results captured

side-effects

every declared side effect validated (only when declared)

never

coverage reports how much of the trace holds the evidence: calls with an outcome, a duration and a cost, and whether the final answer is known.

toolscore.evidence.STATES: tuple[str, ...] = ('pass', 'fail', 'unknown')

The three states, in report order.

class toolscore.evidence.Check(check, status, detail)[source]

One check’s verdict.

Variables:

  • check (str) – The check ID ("required-calls"…).

  • status (str) – "pass", "fail" or "unknown".

  • detail (str) – Why, in a few words.

Parameters:
check: str
status: str
detail: str
__init__(check, status, detail)
Parameters:
Return type:

None

toolscore.evidence.tally(statuses)[source]

Count statuses: {"pass": n, "fail": n, "unknown": n} (always all three keys).

Return type:

dict[str, int]

Parameters:

statuses (list[str])

toolscore.evidence.summary_line(counts)[source]

"5 pass, 1 fail, 2 unknown".

Return type:

str

Parameters:

counts (dict[str, int])

toolscore.evidence.tri_state(value)[source]

True -> pass, False -> fail, None -> unknown.

Return type:

str

Parameters:

value (bool | None)

toolscore.evidence.evaluation_evidence(result)[source]

The tri-state checks and evidence coverage of one evaluation.

Parameters:

result (EvaluationResult) – The evaluation.

Return type:

dict[str, Any]

Returns:

checks (check, status, detail dicts), counts (pass/fail/unknown) and coverage: calls, with_outcome, with_duration, with_cost (counts of calls) and final_answer ("known"/"unknown").

Statistics and selection

See Comparing Runs.

toolscore.stats.wilson_interval(successes, n, z=1.959963984540054)[source]

The Wilson score interval for a proportion.

Parameters:

  • successes (int) – Successes.

  • n (int) – Trials (0 gives (0.0, 1.0): nothing is known).

  • z (float) – Normal quantile (1.96 for 95%).

Return type:

tuple[float, float]

Returns:

(lower, upper) in [0, 1].

toolscore.stats.fisher_exact(a, b, c, d)[source]

Two-sided Fisher exact test p-value for the 2x2 table [[a, b], [c, d]].

Rows are variants, columns pass / not pass. The p-value sums the probabilities of every table with the same margins that is no more likely than the observed one.

Return type:

float

Returns:

The p-value in [0, 1].

Parameters:
toolscore.stats.pass_hat_k(passes, n, k)[source]

Unbiased estimate of the probability that k runs of a task all pass.

C(passes, k) / C(n, k); None when n < k (not enough runs).

Return type:

float | None

Parameters:
toolscore.stats.compare_runs(runs, k=1, baseline=None, min_n=10, alpha=0.05)[source]

Compare variants over repeated runs (see the module docstring).

Parameters:

  • runs (list[Run]) – Every run, of every variant.

  • k (int) – The k of pass^k.

  • baseline (str | None) – Compare every variant with this one; None compares all pairs.

  • min_n (int) – Runs per variant needed before a difference can be called.

  • alpha (float) – Significance level for the Fisher test.

Return type:

Comparison

Returns:

The Comparison.

Raises:

ValueError – If baseline is not one of the variants.

toolscore.stats.load_runs(paths, variant_key=None)[source]

Read runs from JSONL files (one JSON object per line; blank lines skipped).

Raises:

ValueError – On a line that is not a JSON object, naming the file and line.

Return type:

list[Run]

Parameters:

  • paths (Iterable[str | Path])

  • variant_key (str | None)

toolscore.stats.runs_from_traces(variant, directory, gold, min_score=0.9, task_pattern='^(?P<task>.+?)(?:[-_.]r?\\d+)?$')[source]

Score every *.json trace in a directory against a gold file.

A trace passes when its score is at least min_score and it has no error-severity honesty finding; a trace that cannot be read is an error (it stays in the population).

Parameters:

  • variant (str) – The variant name.

  • directory (str | Path) – The directory of trace files.

  • gold (str | Path) – The gold file or contract.

  • min_score (float) – The passing score.

  • task_pattern (str) – Regex with a task group applied to each file stem.

Return type:

list[Run]

Returns:

One run per trace file, sorted by file name.

toolscore.stats.run_from_row(row, variant_key=None)[source]

Read a run from a JSONL row: a toolscore agent run row or an exported record.

The variant is row[variant_key], else arm_id, variant, arm or model; the task task_id or task. The outcome is outcome, else derived: status == "error" is an error, passed true/false is a pass/fail, anything else unknown. Cost is cost_usd or cost (a number, or {"usd": ...}); latency duration_s, latency or duration.

Raises:

ValueError – If the row names no variant.

Return type:

Run

Parameters:
toolscore.stats.select_variant(comparison, policy='lexicographic', objectives=None, weights=None)[source]

Pick a variant.

  • lexicographic: best on the first objective, ties broken by the next.

  • weighted: each objective’s values are scaled to [0, 1] across the variants (1 is best; unknown is 0), and the weighted sum is highest.

  • pareto: the Pareto front; when it has several variants, the first objective breaks the tie (then the next).

Parameters:
Return type:

Selection

Returns:

The Selection.

Raises:

ValueError – On an unknown policy or metric.

toolscore.stats.pareto_front(variants, objectives)[source]

Variants no other variant dominates (at least as good on every objective, better on one).

Unknown values count as the worst.

Return type:

list[str]

Parameters:
toolscore.stats.evaluate_protocol(protocol, comparison)[source]

Evaluate decision rules declared before the runs.

Protocol keys: protocol_version (1), title, baseline, candidate, alpha (0.05), min_runs (runs per variant before any rule can pass; default the comparison’s min_n) and rules. Each rule has an id, a metric (OBJECTIVE_METRICS) and one of:

  • expect: candidate_better: the candidate is better than the baseline (for pass_rate: distinguishable by the Fisher test and the intervals; for other metrics: better by more than tolerance);

  • expect: not_worse: the candidate is not worse by more than tolerance (default 0);

  • min / max: bounds on the candidate’s value.

A rule whose metric is unknown, or whose variants have fewer than min_runs runs, is unknown, never a pass.

Raises:

ValueError – On unknown keys, variants or metrics.

Return type:

ProtocolResult

Parameters:
toolscore.stats.load_protocol(path)[source]

Read a protocol file (YAML with pip install "tool-scorer[yaml]", or JSON).

Return type:

dict[str, Any]

Parameters:

path (str | Path)

class toolscore.stats.Run(variant, task, outcome, cost=None, latency=None, run_id='')[source]

One run of one task under one variant.

Variables:

  • variant (str) – The variant (arm, model, configuration…).

  • task (str) – The task.

  • outcome (str) – "pass", "fail", "error" or "unknown".

  • cost (float | None) – Cost in USD, or None when unknown.

  • latency (float | None) – Wall-clock seconds, or None when unknown.

  • run_id (str) – An identifier for reports.

Parameters:
variant: str
task: str
outcome: str
cost: float | None = None
latency: float | None = None
run_id: str = ''
__init__(variant, task, outcome, cost=None, latency=None, run_id='')
Parameters:
Return type:

None

class toolscore.stats.VariantStats(variant, n, counts, pass_rate, interval, pass_hat_k, k, tasks, tasks_short, cost, cost_known, latency, latency_known)[source]

The summary of one variant (see the module docstring).

Variables:

  • variant (str) – The variant.

  • n (int) – Runs.

  • counts (dict[str, int]) – Runs per outcome.

  • pass_rate (float | None) – pass / n (None with no runs).

  • interval (tuple[float, float]) – 95% Wilson interval of the pass rate.

  • pass_hat_k (float | None) – Mean pass^k over tasks with at least k runs, or None.

  • k (int) – The k used.

  • tasks (int) – Tasks seen.

  • tasks_short (int) – Tasks with fewer than k runs (left out of pass^k).

  • cost (float | None) – Mean cost of the runs that have one, or None.

  • cost_known (int) – Runs with a cost.

  • latency (float | None) – Mean latency of the runs that have one, or None.

  • latency_known (int) – Runs with a latency.

Parameters:
variant: str
n: int
counts: dict[str, int]
pass_rate: float | None
interval: tuple[float, float]
pass_hat_k: float | None
k: int
tasks: int
tasks_short: int
cost: float | None
cost_known: int
latency: float | None
latency_known: int
metric(name)[source]

A metric for selection (see OBJECTIVE_METRICS).

Return type:

float | None

Parameters:

name (str)

__init__(variant, n, counts, pass_rate, interval, pass_hat_k, k, tasks, tasks_short, cost, cost_known, latency, latency_known)
Parameters:
Return type:

None

class toolscore.stats.Comparison(variants, pairs, k, min_n, alpha, notes=list())[source]

The result of compare_runs().

Variables:

  • variants (list[VariantStats]) – Per-variant statistics, in first-seen order.

  • pairs (list[PairComparison]) – Pairwise comparisons (each variant against the baseline, or all pairs).

  • k (int) – The k of pass^k.

  • min_n (int) – Runs needed per variant before a difference can be called.

  • alpha (float) – Significance level.

Parameters:
variants: list[VariantStats]
pairs: list[PairComparison]
k: int
min_n: int
alpha: float
notes: list[str]
by_name()[source]

Statistics by variant name.

Return type:

dict[str, VariantStats]

to_dict()[source]

A JSON-safe report.

Return type:

dict[str, Any]

__init__(variants, pairs, k, min_n, alpha, notes=<factory>)
Parameters:
Return type:

None

class toolscore.stats.PairComparison(a, b, difference, p_value, overlap, distinguishable, verdict)[source]

Two variants side by side.

Variables:

  • a (str) – The first variant.

  • b (str) – The second variant.

  • difference (float | None) – pass_rate(b) - pass_rate(a), or None.

  • p_value (float) – Two-sided Fisher exact p-value.

  • overlap (bool) – Whether the 95% intervals overlap.

  • distinguishable (bool) – Whether the difference is distinguishable (see the module docstring).

  • verdict (str) – A sentence for reports.

Parameters:
a: str
b: str
difference: float | None
p_value: float
overlap: bool
distinguishable: bool
verdict: str
__init__(a, b, difference, p_value, overlap, distinguishable, verdict)
Parameters:
Return type:

None

class toolscore.stats.Selection(chosen, policy, objectives, ranking, pareto_front, scores, unknown, reason)[source]

The result of select_variant().

Variables:

  • chosen (str | None) – The selected variant, or None with no variants.

  • policy (str) – "lexicographic", "weighted" or "pareto".

  • objectives (list[Objective]) – The objectives used.

  • ranking (list[str]) – Variants best first, under the policy.

  • pareto_front (list[str]) – The non-dominated variants.

  • scores (dict[str, float]) – Weighted scores (weighted policy only).

  • unknown (dict[str, list[str]]) – Per variant, the objectives whose value is unknown.

  • reason (str) – Why the chosen variant won.

Parameters:
chosen: str | None
policy: str
objectives: list[Objective]
ranking: list[str]
pareto_front: list[str]
scores: dict[str, float]
unknown: dict[str, list[str]]
reason: str
to_dict()[source]

A JSON-safe report.

Return type:

dict[str, Any]

__init__(chosen, policy, objectives, ranking, pareto_front, scores, unknown, reason)
Parameters:
Return type:

None

class toolscore.stats.ProtocolResult(title, protocol_sha256, baseline, candidate, rules, decision)[source]

The result of evaluate_protocol().

Variables:

  • title (str) – The protocol’s title.

  • protocol_sha256 (str) – Hash of the protocol as declared (record it before the runs).

  • baseline (str) – The baseline variant.

  • candidate (str) – The candidate variant.

  • rules (list[RuleOutcome]) – Each rule’s outcome.

  • decision (str) – "adopt" (every rule passed), "reject" (a rule failed) or "undecided" (a rule is unknown and none failed).

Parameters:

  • title (str)

  • protocol_sha256 (str)

  • baseline (str)

  • candidate (str)

  • rules (list[RuleOutcome])

  • decision (str)

title: str
protocol_sha256: str
baseline: str
candidate: str
rules: list[RuleOutcome]
decision: str
to_dict()[source]

A JSON-safe report.

Return type:

dict[str, Any]

__init__(title, protocol_sha256, baseline, candidate, rules, decision)
Parameters:

  • title (str)

  • protocol_sha256 (str)

  • baseline (str)

  • candidate (str)

  • rules (list[RuleOutcome])

  • decision (str)

Return type:

None

Export

toolscore.export.export_records(inputs, gold=None, variant='', min_score=0.9)[source]

Run records for agent-run JSONL files, trace files and receipts.

Parameters:

  • inputs (Iterable[str | Path]) – .jsonl files from toolscore agent run, trace files (any supported format), or receipts from toolscore eval --receipt.

  • gold (str | Path | None) – A gold file or contract for traces (receipts carry their own).

  • variant (str) – The variant of traces and receipts.

  • min_score (float) – The passing score of traces and receipts.

Return type:

list[dict[str, Any]]

Returns:

The records, in input order.

Raises:

ValueError – On an unreadable input.

toolscore.export.write_export(path, records, inputs)[source]

Write records as JSONL and a manifest next to it.

Return type:

tuple[Path, Path]

Returns:

(jsonl path, manifest path). The manifest holds the schema, the producer, the sources with their hashes, the record count and the SHA-256 of the JSONL file.

Parameters:
toolscore.export.validate_record(record)[source]

Check a record against the shipped JSON Schema (Toolscore’s conservative validator).

Return type:

list[str]

Returns:

Problems found (empty when the record conforms), plus a check that hashes.record_sha256 matches the record.

Parameters:

record (dict[str, Any])

toolscore.export.record_from_run(row, source)[source]

The run record of one toolscore agent run row.

Parameters:

  • row (dict[str, Any]) – The row.

  • source (dict[str, Any]) – {"kind", "path", "sha256", "line"} of where it came from.

Return type:

dict[str, Any]

Returns:

The record (see the module docstring and SCHEMA_PATH).

toolscore.export.SCHEMA_VERSION = '1'

The schema version of a run record.

Doctor

toolscore.doctor.run_doctor(spec_path=None, contract_path=None, server=None, keys=None, allow_unknown_price=False, start_server=True)[source]

Run the preflight checks (see the module docstring).

Parameters:

  • spec_path (str | Path | None) – An agent spec to check (server, keys, packages, prices, budget).

  • contract_path (str | Path | None) – A contract or gold file to validate.

  • server (MCPServerSpec | None) – A server to start and list (when there is no spec).

  • keys (list[str] | None) – More environment variable names that must be set.

  • allow_unknown_price (bool) – As for toolscore agent run.

  • start_server (bool) – Start the spec’s server (False checks only the files).

Return type:

list[DoctorCheck]

Returns:

The checks, in the order they ran.

toolscore.doctor.check_server(spec, verify=None, timeout=30.0)[source]

Start a server, list its tools and check its build; nothing else is called.

Return type:

list[DoctorCheck]

Parameters:
toolscore.doctor.doctor_report(checks)[source]

A JSON-safe report: the checks, the counts and the worst status.

Return type:

dict[str, Any]

Parameters:

checks (list[DoctorCheck])

class toolscore.doctor.DoctorCheck(name, status, detail)[source]

One preflight check: name, status and a detail sentence.

Parameters:
name: str
status: str
detail: str
__init__(name, status, detail)
Parameters:
Return type:

None