For agent builders
Test your agent
Snapshot your agent's tool calls and fail CI the instant a prompt or model change makes it call the wrong tool, with the wrong arguments, in the wrong order. No hand-written expected-call files, no YAML.
Snapshot testing
Record → approve → replay
Stop hand-writing expected tool calls. Record your agent's behavior once, approve it, and replay it forever — Jest snapshots, for agents. Your agent's own behavior becomes the regression test.
toolscore init
detects your framework and scaffolds a passing pytest suite plus a CI workflow. The
toolscore_snapshot
fixture ships with the package — no plugin registration.
pip install tool-scorer
toolscore init # detects your framework, scaffolds a passing pytest suite
pytest # first run RECORDS your agent's tool calls as snapshots
toolscore approve --all # review, then approve them as the baseline
pytest # every run after this REPLAYS — and fails on driftRecord
The first pytest run captures your agent’s tool calls into pending snapshots and passes with a warning. No hand-written expected calls, no YAML.
Approve
Review the recorded calls, then bless them as the baseline. Snapshots are plain JSON under .toolscore/snapshots/ — they review cleanly in PRs.
Replay
Every run after that replays against the baseline. Drift fails the test with a full expected-vs-actual diff. Re-record on purpose with --toolscore-update.
def test_books_a_flight(toolscore_snapshot):
toolscore_snapshot(my_agent("book a flight to NYC"))
# First run: records a pending snapshot and warns.
# After `toolscore approve`: replays against the baseline, fails on drift.Or evaluate a trace
A–F grade + the same "Top issues to fix"
Already have a captured trace and a gold standard? toolscore eval scores it deterministically and gives you the same graded verdict — selection accuracy,
argument F1, sequence accuracy, and redundancy, blended into one score.
toolscore eval gold.json trace.json Expected vs Actual Tool Calls
┏━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓
┃ # ┃ Expected ┃ Actual ┃ Status ┃
┡━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩
│ 1 │ search_flights(dest= │ search_flights(dest= │ destination: │
│ │ 'NYC') │ 'BOS') │ 'NYC' ≠ 'BOS' │
│ 2 │ book_flight(...) │ cancel_booking(...) │ tool: 'book_flight' │
│ │ │ │ ≠ 'cancel_booking' │
└─────┴──────────────────────┴──────────────────────┴──────────────────────┘
score 0.47 < 0.90 required · selection 0.50 · args 0.40 · sequence 0.50Prefer explicit expectations?
The fluent expect() API
It reads like the sentence you'd say out loud. Chain
.calls(),
.then_calls(), and
.does_not_call() — and loosen exactly
the arguments you choose with matchers.
from toolscore import expect, ANY, Regex
expect(agent).on("book me a flight to NYC") \
.calls("search_flights", origin=ANY, destination="NYC") \
.then_calls("book_flight", flight_id=Regex(r"FL-\d+")) \
.does_not_call("cancel_booking") \
.with_score(0.9) \
.run()Matchers — flexible where it matters
| Matcher | Matches | Example |
|---|---|---|
| ANY | anything | calls("search", q=ANY) |
| Regex(pattern) | full string match | Regex(r"FL-\d+") |
| Approx(value, rel, abs) | numbers within tolerance | Approx(40.71, rel=1e-2) |
| Contains(item) | membership in str/list/dict | Contains("metric") |
| OneOf(*values) | any of the candidates | OneOf("NYC", "New York") |
| IsType(*types) | isinstance check (bool-safe) | IsType(int) |
Native everywhere — zero glue
Pass raw responses straight in
Toolscore auto-detects the format — no manual extraction. The same is true in
evaluate(),
expect(), and the snapshot fixture.
Plus pytest-native fixtures, @toolscore.cases()
data-driven tests, async support, and a CI gate — all in one package.
Make your agent's behavior a regression test
pip install tool-scorer, then toolscore init. Record once, approve, and replay forever.
pip install tool-scorer uvx tool-scorer demo