Skip to content

MCP servers · agents · deterministic, offline, $0

The instant health-check for LLM tool-calling

Point Toolscore at an MCP server or an agent and get a clear "here’s your grade and exactly what’s broken" verdict — deterministically, offline, with zero API cost. No LLM judge, no cloud, no per-test bill.

Install

$ pip install tool-scorer

Or run instantly, zero install

$ uvx tool-scorer demo
Quickstart Test an MCP server Test an agent →
deterministic runs offline $0 per run pytest-native
zsh — toolscore live demo
$
B
MCP Scorecard
notes-server 1.0.0
Grade B · Score 87%
happy 80%
edge 100%
lint 93%
Tools
Tool Scenarios Avg latency Def. tokens
create_note 6/6 0.1 ms 80
list_notes 6/6 0.1 ms 59
search_notes 6/6 0.1 ms 48
delete_note 6/6 0.1 ms 52
export_notes 3/6 0.1 ms 64
Tool definitions cost ~303 estimated tokens of context across 5 tool(s).
Top issues to fix
  1. 1. export_notes fails on valid input (export failed: storage backend not configured)
    -> The tool errors on well-formed arguments — check the handler and the input schema.
  2. 2. delete_note property 'note_id' is missing a 'type'
    -> Give the property a JSON-schema type (and an enum where values are fixed).
  3. 3. search_notes description is very short (< 10 chars)
    -> Describe what the tool does and when to use it.

Two sides of one handshake

Pick your side of the LLM ↔ tool handshake

Toolscore tests both ends of the same contract between a model and a tool — whether you ship the tool, or build the agent that calls it.

You ship an MCP server

Grade your MCP server

Toolscore runs your server through generated happy-path and adversarial edge-case scenarios, lints the schemas, measures token cost, and grades whether an LLM can actually use it — before you publish.

$ toolscore mcp test "python my_server.py"
  • ✓ Happy-path + edge-case scenarios, auto-generated
  • ✓ Schema lint + token-cost signal
  • ✓ A–F grade with "Top issues to fix"
Test your MCP server →

You build an agent

Test your agent

Snapshot your agent’s tool calls and fail CI the instant a prompt or model change makes it call the wrong tool, with the wrong arguments, in the wrong order. Jest snapshots, for agents.

$ toolscore init # record -> approve -> replay
  • ✓ Snapshot record / approve / replay
  • ✓ Fluent expect() API + matchers
  • ✓ Auto-detects 8 agent frameworks
Test your agent →

How it works

From a launch command to a graded verdict

Four steps, all offline. No model in the loop by default, no API keys, no per-test cost.

  1. 01

    Point it at a target

    An MCP server launch command, or your agent’s tool calls. No glue code — raw OpenAI / Anthropic / Gemini / framework responses are auto-detected.

    toolscore mcp test "python my_server.py"
  2. 02

    It generates scenarios

    From each tool’s JSON schema, Toolscore builds happy-path and adversarial edge-case inputs, then lints the definitions and estimates token cost.

    6 scenarios / tool · happy + edge · schema lint
  3. 03

    It runs everything offline

    Deterministic, local, zero API cost. No LLM judge by default, no cloud, no per-test bill. The same input always yields the same grade.

    offline · $0 per run · deterministic
  4. 04

    You get a graded verdict

    An A–F scorecard, a per-tool table, and a ranked "Top issues to fix" list with concrete suggestions. Gate CI with --fail-under or --ci.

    Grade B (87%) · Top issues to fix →

What you get

Built to live in your test suite

Not a dashboard you check after the fact — a deterministic gate that fails the build the moment tool-calling breaks.

Top issues to fix

Not just a number — a ranked, plain-English list of what is broken and how to fix it, from broken handlers to untyped schemas to thin descriptions.

Token-cost signal

Every tool definition costs context. Toolscore estimates per-tool definition tokens so you can trim bloat before it taxes every single request.

Zero LLM cost, deterministic

All core metrics run offline with no LLM judge. The same inputs always produce the same grade — so it belongs in CI, not a billing dashboard.

CI gate built in

--fail-under B exits non-zero below the bar; --ci writes the verdict straight to your GitHub Actions job summary and fails on blocking issues.

Framework auto-detect

Pass raw responses from OpenAI, Anthropic, Gemini, LangGraph, Pydantic AI, OpenAI Agents SDK, Claude Agent SDK, or CrewAI. Toolscore detects the format.

Snapshot record / replay

Record your agent’s tool calls once, approve them as the baseline, and replay forever. Drift fails the build with a full expected-vs-actual diff.

Toolscore vs the field

The deterministic, in-CI option

Observability and LLM-eval platforms watch your agent in production. Toolscore is the free, deterministic health-check that runs in CI and fails the build on drift. Here is where it fits.

See the full matrix →
Capability Toolscore DeepEval LangChain agentevals EvalView mcp-eval
Deterministic — no LLM judge required Yes Partial Yes Partial Partial
Runs fully offline ($0 per run) Yes No Yes No No
Snapshot record / approve / replay Yes No No No No
MCP-server scorecard (A–F + lint + token cost) Yes No No No Partial
Ranked "Top issues to fix" verdict Yes No No Partial No
pytest-native (drop-in fixture + assertions) Yes Partial No No No
Framework auto-detect (8 SDKs) Yes Partial Partial No No
CI gate built in (--fail-under / --ci) Yes Partial No No Partial
✓ built-in ~ partial / via add-on – not a focus

Grade your tool-calling in seconds

Install it, point it at a server or an agent, and read the verdict. Free, deterministic, offline.

$ pip install tool-scorer
$ uvx tool-scorer demo
Read the quickstart Star on GitHub View on PyPI →