Adapters Module

Adapters convert various LLM trace formats into a normalized format for evaluation.

Base Classes

class toolscore.adapters.ToolCall(tool, args=None, result=None, timestamp=None, duration=None, cost=None, metadata=<factory>)[source]

Represents a single tool call in a trace.

Variables:

tool – Name of the tool/function called.
args – Arguments provided to the tool. None carries a specific meaning for gold/expected calls: “do not check arguments” — the tool name must match but its arguments are ignored. For actual/trace calls, None simply means no arguments were recorded and is treated as an empty mapping. Adapters that parse real traces always set a concrete dict, so None in practice only originates from gold specifications that omit args.
result – Result returned by the tool (optional).
timestamp – Unix timestamp of when the call was made (optional).
duration – Duration of the call in seconds (optional).
cost – Cost associated with this call in USD (optional).
metadata – Additional metadata about the call. Failures are recorded as metadata["is_error"] and/or metadata["error"] (the message); see is_error.

Parameters:

tool (str)
args (dict[str, Any] | None)
result (Any)
timestamp (float | None)
duration (float | None)
cost (float | None)
metadata (dict[str, Any])

tool: str

args: dict[str, Any] | None = None

result: Any = None

timestamp: float | None = None

duration: float | None = None

cost: float | None = None

metadata: dict[str, Any]

__post_init__()[source]

Validate tool call data after initialization.

args is intentionally not coerced from None to {} so that gold calls can express “do not check arguments” (args is None) distinctly from “expect exactly zero arguments” (args == {}). Consumers that need a concrete mapping use call.args or {}.

Return type:: None

property is_error: bool

Whether the call failed.

Read from metadata["is_error"] or a non-empty metadata["error"], which is how the MCP and custom adapters and toolscore.evaluate() record failures. A call with no error information is not an error.

__init__(tool, args=None, result=None, timestamp=None, duration=None, cost=None, metadata=<factory>)

Parameters:

tool (str)
args (dict[str, Any] | None)
result (Any)
timestamp (float | None)
duration (float | None)
cost (float | None)
metadata (dict[str, Any])

Return type:

None

class toolscore.adapters.BaseAdapter[source]

Abstract base class for trace format adapters.

All trace format adapters must inherit from this class and implement the parse method to convert provider-specific formats into a normalized list of ToolCall objects.

abstractmethod parse(trace_data)[source]

Parse trace data into a normalized list of tool calls.

Parameters:: trace_data (dict[str, Any] | list[Any]) – The raw trace data from the LLM provider.
Return type:: list[ToolCall]
Returns:: A list of ToolCall objects in chronological order.
Raises:: ValueError – If the trace data is invalid or cannot be parsed.

Adapter Implementations

OpenAI Adapter

class toolscore.adapters.OpenAIAdapter[source]

Bases: BaseAdapter

Adapter for OpenAI function call traces.

Parses OpenAI Chat Completion API conversation logs that include function/tool calls in the message history.

parse(trace_data)[source]

Parse OpenAI trace into normalized tool calls.

Parameters:: trace_data (dict[str, Any] | list[Any]) – OpenAI message history or response containing function calls. Can be a list of messages or a dict with ‘messages’ key.
Return type:: list[ToolCall]
Returns:: List of ToolCall objects extracted from the trace.
Raises:: ValueError – If trace format is invalid.

Anthropic Adapter

class toolscore.adapters.AnthropicAdapter[source]

Bases: BaseAdapter

Adapter for Anthropic Claude tool-use traces.

Parses Anthropic/Claude API conversation logs that include tool_use content blocks in assistant messages.

parse(trace_data)[source]

Parse Anthropic trace into normalized tool calls.

Parameters:: trace_data (dict[str, Any] | list[Any]) – Anthropic message history containing tool_use blocks. Can be a list of messages or a dict with ‘messages’ key.
Return type:: list[ToolCall]
Returns:: List of ToolCall objects extracted from the trace.
Raises:: ValueError – If trace format is invalid.

LangChain Adapter

class toolscore.adapters.LangChainAdapter[source]

Bases: BaseAdapter

Adapter for LangChain agent traces.

Supports parsing of: - AgentAction objects (legacy format) - ToolCall objects (modern format) - Raw dictionaries with tool/action information

Example LangChain trace formats:

Legacy format (AgentAction):

[
    {
        "tool": "search",
        "tool_input": {"query": "Python"},
        "log": "Invoking search..."
    }
]

Modern format (ToolCall):

[
    {
        "name": "search",
        "args": {"query": "Python"},
        "id": "call_123"
    }
]

parse(data)[source]

Parse LangChain trace data.

Parameters:: data (Any) – LangChain trace data (list of actions/calls)
Return type:: list[ToolCall]
Returns:: List of ToolCall objects
Raises:: ValueError – If data format is invalid

Supports both legacy (AgentAction) and modern (ToolCall) LangChain formats.

Gemini Adapter

class toolscore.adapters.GeminiAdapter[source]

Bases: BaseAdapter

Adapter for Google Gemini function call traces.

Parses Google Gemini API conversation logs that include function calls in the candidate responses.

parse(trace_data)[source]

Parse Gemini trace into normalized tool calls.

Parameters:: trace_data (dict[str, Any] | list[Any]) – Gemini message history or response containing function calls. Can be a list of messages or a dict with ‘candidates’ key.
Return type:: list[ToolCall]
Returns:: List of ToolCall objects extracted from the trace.
Raises:: ValueError – If trace format is invalid.

MCP Adapter

class toolscore.adapters.MCPAdapter[source]

Bases: BaseAdapter

Adapter for Anthropic Model Context Protocol (MCP) traces.

MCP is an open standard for connecting AI assistants to data systems using JSON-RPC 2.0 messaging format.

Supports:

Tool call requests (JSON-RPC 2.0 method calls)
Tool call results (JSON-RPC 2.0 responses)
Error handling (JSON-RPC 2.0 errors)
Both single requests and batch requests

A recorded session (for example from toolscore mcp record) holds each request followed by its response. A response whose id matches an earlier request in the same trace is merged into that request’s call (result, error, is_error), so one tool call stays one ToolCall. A response with no matching request is reported on its own, as before.

Example MCP tool call request:

{
    "jsonrpc": "2.0",
    "method": "tools/call",
    "params": {
        "name": "get_weather",
        "arguments": {"location": "San Francisco"}
    },
    "id": 1
}

Example MCP tool call result:

{
    "jsonrpc": "2.0",
    "id": 1,
    "result": {
        "content": [{"type": "text", "text": "Temperature: 72°F"}]
    }
}

parse(trace_data)[source]

Parse MCP trace into normalized tool calls.

Parameters:: trace_data (dict[str, Any] | list[Any]) – MCP trace data (single request or list of requests).
Return type:: list[ToolCall]
Returns:: List of normalized ToolCall objects.
Raises:: ValueError – If trace data is invalid.

Reads JSON-RPC 2.0 message lists and sessions written by toolscore mcp record. Each tools/call response is paired with its request by id, so the call keeps its result, error and duration.

OpenTelemetry Adapter

class toolscore.adapters.OTelAdapter[source]

Bases: BaseAdapter

Adapter for OpenTelemetry GenAI tool spans (OTLP JSON exports).

See tool_calls_from_otel() for the accepted inputs and the mapping.

parse(trace_data)[source]

Parse an OTLP export or span list into tool calls.

Parameters:: trace_data (dict[str, Any] | list[Any]) – An OTLP JSON export or a list of spans.
Return type:: list[ToolCall]
Returns:: One ToolCall per tool span, in start-time order.

toolscore.adapters.otel.tool_calls_from_otel(data)[source]

Extract tool calls from OpenTelemetry GenAI tool spans.

Parameters:: data (Any) – An OTLP JSON export, a list of span dicts, or a list of OpenTelemetry SDK span objects.
Return type:: list[dict[str, Any]]
Returns:: One dict per tool span, in start-time order, with tool, args, result, is_error, error, duration (seconds) and id. Arguments and results recorded as JSON strings are decoded; arguments that are not a JSON object are kept as {"value": ...}.

Custom Adapter

class toolscore.adapters.CustomAdapter[source]

Bases: BaseAdapter

Adapter for custom/generic JSON trace formats.

Accepts a simplified trace format with a ‘calls’ array or directly as an array of tool call objects.

parse(trace_data)[source]

Parse custom JSON trace into normalized tool calls.

Parameters:: trace_data (dict[str, Any] | list[Any]) – Custom trace format. Can be: - {“calls”: […]} with array of call objects - Direct array of call objects Each call object should have at minimum a ‘tool’ or ‘name’ field.
Return type:: list[ToolCall]
Returns:: List of ToolCall objects extracted from the trace.
Raises:: ValueError – If trace format is invalid.