Toolscore Documentation

Toolscore is a Python package for evaluating LLM tool usage against gold standard specifications. It helps developers benchmark different models, validate agent behavior, and track improvements in function calling accuracy over time.

What is Toolscore?

Toolscore evaluates LLM tool usage - it doesn’t call LLM APIs directly. Think of it as a testing framework for function-calling agents:

✅ Evaluates tool usage traces from OpenAI, Anthropic, Gemini, agent frameworks, recorded MCP sessions, OpenTelemetry spans, or custom sources

✅ Compares actual behavior against expected gold standards

✅ Reports detailed metrics on accuracy, efficiency, and correctness, plus failed calls, leaked credentials and forbidden calls

✅ Grades and lints MCP servers, including references to missing tools and tool poisoning

❌ Does NOT call LLM APIs or execute tools (you capture traces separately)

Quick Start

pip install tool-scorer

# Run evaluation
toolscore eval examples/gold_calls.json examples/trace_openai.json --html report.html

# Grade an MCP server
toolscore mcp test "python my_server.py"

Key Features

• Comprehensive Metrics Suite: Tool invocation accuracy, selection accuracy, sequence edit distance, argument matching, redundant and repeated calls, required calls completed, and side-effect validation

• Behavior and Safety Checks: Failed calls, blind retries, credentials passed into tool arguments, and forbidden-call policies, with a CI gate (Behavior and Safety Checks)

• Snapshot Testing: Record, approve and replay your agent’s tool calls in pytest (Snapshot Testing)

• MCP Scorecard, Lint and Recorder: Grade any MCP server, lint it for missing-tool references and tool poisoning, and record real sessions (Testing MCP Servers)

• Native Everywhere: OpenAI, Anthropic, Gemini, LangGraph, Pydantic AI, OpenAI Agents SDK, Claude Agent SDK, CrewAI, MCP and OpenTelemetry traces (Framework Integration Guide)

• CLI, Python API and GitHub Action: Command-line interface, programmatic usage and CI gates

• Rich Reports: Console, HTML, Markdown, CSV and machine-readable JSON reports, plus a versioned record for evaluation harnesses

• Extensible: Easy to add custom metrics and validators

Contents

Getting Started

• Installation
  • Requirements
  • From PyPI
  • From Source
  • Development Installation
  • Optional Dependencies
  • Verification
  • Next Steps
• Quick Start
  • Install Toolscore
  • The fastest path: toolscore init + snapshots
  • Evaluating captured traces
  • Basic Usage
  • Creating Gold Standards
  • Supported Trace Formats
  • Check behavior and safety
  • Next Steps

Guides

• User Guide
  • Understanding Metrics
  • Scoring Semantics
  • Working with Trace Formats
  • Capturing Traces
  • Creating Effective Gold Standards
  • Side-Effect Validation
  • Generating Reports
  • Batch Evaluation
  • End-to-End Agent Testing
  • Data-Driven Testing with @toolscore.cases()
  • Pytest Integration
  • Interactive Tutorials
  • Tips and Tricks
  • Troubleshooting
• Snapshot Testing
  • The record → approve → replay story
  • The toolscore_snapshot fixture
  • CLI commands
  • Snapshot file format
  • CI behavior
  • The update flow
  • Programmatic API
• Argument Matchers
  • The matchers
  • The omitted-args vs {} contract
  • Strict mode interplay
  • API reference
• Framework Integration Guide
  • LangGraph
  • Pydantic AI
  • OpenAI Agents SDK
  • Claude Agent SDK
  • CrewAI (experimental)
  • OpenTelemetry (any instrumented framework)
  • MCP sessions
• Fluent API — expect()
  • Hero Example
  • Quick Start
  • Chain-Method Reference
  • Using Matchers
  • Async Agents
  • Forbidden-Only Contract
  • API Reference
• Behavior and Safety Checks
  • Failed calls and blind retries
  • Required calls completed
  • Credentials in tool arguments
  • Forbidden calls
  • Failing a build on violations
  • A complete record for harnesses
• LLM-as-a-Judge
  • JudgeConfig
  • Provider inference
  • Environment-variable keys
  • Local / OpenAI-compatible endpoints (Ollama, vLLM, Groq)
  • Batching behavior
  • Extras install matrix
  • Using the judge from evaluate_trace
  • CLI flags
  • API reference
• Testing MCP Servers
  • Test your MCP server in 60 seconds
  • The subcommands
  • What the linter checks
  • Recording real sessions
  • What the grade means
  • Tuning the run
  • Using it in CI
  • Embedding the scorecard
  • Python API
• Extending Toolscore
  • When you need a custom adapter
  • The BaseAdapter contract
  • Using a custom adapter
  • Request an adapter
• Toolscore vs. Other Tools
  • The short version
  • What the others are
  • Feature comparison
  • When to reach for Toolscore
  • When to reach for a platform instead (or as well)

API Reference

• API Reference
  • Core Module
  • Adapters Module
  • Matchers Module
  • Metrics Module
  • Snapshots Module
  • MCP Module
  • Validators Module
  • Reports Module
  • Verdict Module
  • Overview
  • Basic Usage
  • Main Components
  • Quick Reference
• Core Module
  • Main Functions
  • Result Container
• Adapters Module
  • Base Classes
  • Adapter Implementations
• Matchers Module
  • ANY
  • Approx
  • Contains
  • IsType
  • Matcher
  • OneOf
  • Regex
• Metrics Module
  • Accuracy Metrics
  • Sequence Metrics
  • Argument Metrics
  • Required Calls
  • Efficiency Metrics
  • Safety: Credentials and Forbidden Calls
  • Side-Effect Metrics
  • Performance Metrics
  • LLM-as-a-judge Metrics (Optional)
• Snapshots Module
  • snapshot_check()
  • Snapshot
  • SnapshotStore
• MCP Module
  • Client
  • Recording sessions
  • Server configuration
  • Scorecard harness
  • Scorecard
• Validators Module
  • HTTP Validator
  • Filesystem Validator
  • SQL Validator
• Reports Module
  • JSON Reports
  • HTML Reports
  • Markdown and CSV Reports
  • Behavior and Safety Findings

Development

• Contributing
• Changelog

Indices and tables

• Index

• Module Index

• Search Page