For agent builders
Test your agent
Snapshot your agent's tool calls and fail CI the instant a prompt or model change makes it call the wrong tool, with the wrong arguments, in the wrong order. No hand-written expected-call files, no YAML.
Snapshot testing
Record → approve → replay
Stop hand-writing expected tool calls. Record your agent's behavior once, approve it, and replay it forever — Jest snapshots, for agents. Your agent's own behavior becomes the regression test.
toolscore init
detects your framework and scaffolds a passing pytest suite plus a CI workflow. The
toolscore_snapshot
fixture ships with the package — no plugin registration.
pip install tool-scorer
toolscore init # detects your framework, scaffolds a passing pytest suite
pytest # first run RECORDS your agent's tool calls as snapshots
toolscore approve --all # review, then approve them as the baseline
pytest # every run after this REPLAYS — and fails on driftRecord
The first pytest run captures your agent’s tool calls into pending snapshots and passes with a warning. No hand-written expected calls, no YAML.
Approve
Review the recorded calls, then bless them as the baseline. Snapshots are plain JSON under .toolscore/snapshots/ — they review cleanly in PRs.
Replay
Every run after that replays against the baseline. Drift fails the test with a full expected-vs-actual diff. Re-record on purpose with --toolscore-update.
def test_books_a_flight(toolscore_snapshot):
toolscore_snapshot(my_agent("book a flight to NYC"))
# First run: records a pending snapshot and warns.
# After `toolscore approve`: replays against the baseline, fails on drift.Or evaluate a trace
A–F grade + the same "Top issues to fix"
Already have a captured trace and a gold standard? toolscore eval scores it deterministically and gives you the same graded verdict — selection accuracy,
argument F1, sequence accuracy, and redundancy, blended into one score.
toolscore eval gold.json trace.json Expected vs Actual Tool Calls
┏━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓
┃ # ┃ Expected ┃ Actual ┃ Status ┃
┡━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩
│ 1 │ search_flights(dest= │ search_flights(dest= │ destination: │
│ │ 'NYC') │ 'BOS') │ 'NYC' ≠ 'BOS' │
│ 2 │ book_flight(...) │ cancel_booking(...) │ tool: 'book_flight' │
│ │ │ │ ≠ 'cancel_booking' │
└─────┴──────────────────────┴──────────────────────┴──────────────────────┘
score 0.47 < 0.90 required · selection 0.50 · args 0.40 · sequence 0.50What the score hides
Catch agents that give up after an error
A real run: a small agent asked to read a config file through the official filesystem MCP
server tried a path outside its sandbox, got Access denied, and gave up without
calling list_allowed_directories. Its one
call used the right tool, so the score alone reads 90%. Toolscore shows what actually
happened: neither required call succeeded.
Add --weight required_call_recall=0.3 and this run
grades D. Sessions like this one come from
toolscore mcp record; OpenTelemetry exports work
the same way.
$ toolscore eval gold.json session.json
╭────────────────────────╮
│ Grade B WARN (90.0%) │
╰────────────────────────╯
Required calls completed: 0 of 2 (missing or failed; not part of the default score)
Top issues to fix
1. list_allowed_directories expected call to `list_allowed_directories` never happened
Behavior and safety
WARNING 1 of 1 tool calls failed (read_text_file)Guardrails in CI
Block destructive commands and leaked keys
Every evaluation reports failed calls, retries of a failed call with the same arguments,
and credentials passed into tool arguments (shown redacted). Add rules for calls your agent
must never make — by tool, and by argument with
$regex,
$contains or
$one_of — and fail the build when one
happens.
The same rule in a test:
expect(agent).on(prompt).does_not_call("run_shell", command=Contains("rm -rf"))
$ toolscore eval gold.json trace.json \
--forbidden forbidden.json --fail-on-violations
Behavior and safety
ERROR call 2 run_shell matches forbidden rule 1: destructive shell command
ERROR call 3 http_post passes a credential (aws_access_key_id) in 'body.text': AKIA…LE
WARNING 2 of 5 tool calls failed (deploy x2)
WARNING 1 call repeats a failed call with the same arguments
ERROR 2 forbidden call(s) or credential(s) found (--fail-on-violations) # exit 1Prefer explicit expectations?
The fluent expect() API
It reads like the sentence you'd say out loud. Chain
.calls(),
.then_calls(), and
.does_not_call() — and loosen exactly
the arguments you choose with matchers.
from toolscore import expect, ANY, Regex
expect(agent).on("book me a flight to NYC") \
.calls("search_flights", origin=ANY, destination="NYC") \
.then_calls("book_flight", flight_id=Regex(r"FL-\d+")) \
.does_not_call("cancel_booking") \
.with_score(0.9) \
.run()Matchers — flexible where it matters
| Matcher | Matches | Example |
|---|---|---|
| ANY | anything | calls("search", q=ANY) |
| Regex(pattern) | full string match | Regex(r"FL-\d+") |
| Approx(value, rel, abs) | numbers within tolerance | Approx(40.71, rel=1e-2) |
| Contains(item) | membership in str/list/dict | Contains("metric") |
| OneOf(*values) | any of the candidates | OneOf("NYC", "New York") |
| IsType(*types) | isinstance check (bool-safe) | IsType(int) |
Native everywhere — zero glue
Pass raw responses straight in
Toolscore auto-detects the format — no manual extraction. The same is true in
evaluate(),
expect(), and the snapshot fixture.
Plus pytest-native fixtures, @toolscore.cases()
data-driven tests, async support, and a CI gate — all in one package.
Make your agent's behavior a regression test
pip install tool-scorer, then toolscore init. Record once, approve, and replay forever.
pip install tool-scorer uvx tool-scorer demo