Sessions, Config Lint, Recorders and Inspect AI

Coding-agent sessions

Find and summarize coding-agent sessions on this machine (read-only).

Claude Code and Codex CLI keep every session as a JSON Lines file (toolscore.adapters.sessions has the formats). This module finds them, reads them with the session adapters and runs the honesty checks, so toolscore sessions list can show what a coding agent did yesterday and where it may have reported success after a failure.

Nothing here writes to a session file or sends anything anywhere. export_session() writes a normalized Toolscore trace with credentials redacted, for sharing or for a regression test.

toolscore.sessions.AGENTS = ('claude-code', 'codex'): Agents whose sessions Toolscore can find.

class toolscore.sessions.SessionSummary(path, agent, modified, session_id=None, cwd=None, started=None, calls=0, failed=0, findings=dict(), flagged=None, final_answer=False, error=None)[source]

One session file and what the checks found in it.

Variables:

path (Path) – The session file.
agent (str) – "claude-code" or "codex".
session_id (str | None) – The session ID, when the file records one.
cwd (str | None) – The working directory the agent ran in, when recorded.
started (str | None) – The first timestamp in the file (ISO 8601), when recorded.
modified (datetime) – The file’s modification time.
calls (int) – Tool calls on the main thread.
failed (int) – Calls that failed.
findings (dict[str, int]) – Honesty findings as {rule: count}.
flagged (bool | None) – The honesty verdict (None when unknown).
final_answer (bool) – Whether the session holds a final answer.
error (str | None) – Why the file could not be read, if it could not.

Parameters:

path (Path)
agent (str)
modified (datetime)
session_id (str | None)
cwd (str | None)
started (str | None)
calls (int)
failed (int)
findings (dict[str, int])
flagged (bool | None)
final_answer (bool)
error (str | None)

path: Path

agent: str

modified: datetime

session_id: str | None = None

cwd: str | None = None

started: str | None = None

calls: int = 0

failed: int = 0

findings: dict[str, int]

flagged: bool | None = None

final_answer: bool = False

error: str | None = None

to_json()[source]

A JSON-safe dict of the summary.

Return type:: dict[str, Any]

__init__(path, agent, modified, session_id=None, cwd=None, started=None, calls=0, failed=0, findings=<factory>, flagged=None, final_answer=False, error=None)

Parameters:

path (Path)
agent (str)
modified (datetime)
session_id (str | None)
cwd (str | None)
started (str | None)
calls (int)
failed (int)
findings (dict[str, int])
flagged (bool | None)
final_answer (bool)
error (str | None)

Return type:

None

toolscore.sessions.default_roots()[source]

Where each agent keeps its sessions (honoring CLAUDE_CONFIG_DIR and CODEX_HOME).

Return type:: dict[str, Path]

toolscore.sessions.parse_since(value, now=None)[source]

Parse --since: 30m, 12h, 1d, 2w, or a date / ISO 8601 time.

Parameters:

value (str) – The option value.
now (datetime | None) – The current time (for tests).

Return type:

datetime

Returns:

The earliest modification time to include (timezone-aware).

Raises:

ValueError – If the value cannot be read.

toolscore.sessions.find_session_files(agent='all', root=None, since=None)[source]

Find session files, newest first.

Parameters:

agent (str) – "claude-code", "codex" or "all".
root (Path | None) – Search this directory instead of the agents’ default folders (every *.jsonl under it; the agent is detected from the content).
since (datetime | None) – Only files modified at or after this time.

Return type:

list[tuple[str, Path]]

Returns:

(agent, path) pairs; with root, the agent is "auto".

toolscore.sessions.summarize_session(path, agent='auto')[source]

Read one session file and run the honesty checks on it.

Parameters:

path (Path) – The session file.
agent (str) – The agent, or "auto" to detect it from the content.

Return type:

SessionSummary

Returns:

The summary; error is set when the file is not a session Toolscore reads.

toolscore.sessions.export_session(path, output, max_result_chars=2000)[source]

Write a session as a normalized Toolscore trace, with credentials redacted.

The trace is {"format": "toolscore", "source": ..., "calls": [...], "final_answer": ...}: every call with its tool, arguments, result (cut to max_result_chars), error and ID. Any credential the secret scanner recognizes (API keys, tokens, private keys) is replaced by a redacted preview, in arguments, results and the final answer. Other private content (code, paths) is kept: review the file before you share it.

Parameters:

path (Path) – The session file.
output (Path) – The trace file to write.
max_result_chars (int) – Longest result text kept per call.

Return type:

dict[str, Any]

Returns:

The trace that was written.

toolscore.core.read_trace_data(path)[source]

Read a trace file: one JSON document, or JSON Lines (one value per line).

Coding-agent sessions (Claude Code transcripts, Codex CLI rollouts) and other logs are JSON Lines; they are returned as the list of their lines’ values. Blank lines and lines that are not JSON (a partly written last line, for example) are skipped.

Parameters:: path (str | Path) – The trace file.
Return type:: Any
Returns:: The parsed data.
Raises:: ValueError – If the file is neither JSON nor JSON Lines.

Agent-config lint

See Agent-Config Lint.

Lint agent instruction files and MCP host configs against the tools that really exist.

Agents read instruction files (CLAUDE.md, AGENTS.md, SKILL.md, Cursor rules, Copilot instructions) that name the tools to use. When a server is renamed, a tool is removed, or an instruction tells the agent to delete with a tool that declares itself read-only, nothing warns you: the agent just calls a tool that is not there, or a client lets a “read-only” tool run without asking. lint_config() finds these mismatches.

What it reads:

instruction files: CLAUDE.md, CLAUDE.local.md, AGENTS.md, any SKILL.md, .cursor/rules/**/*.mdc|*.md, .cursorrules, .github/copilot-instructions.md, .github/instructions/*.instructions.md;
MCP host configs: .mcp.json, .cursor/mcp.json, .vscode/mcp.json, claude_desktop_config.json (and, only when asked, the user’s ~/.claude.json);
the tools each server exposes, from snapshot or lock files (toolscore mcp list --json, toolscore mcp pin) or, only when asked, by starting the configured servers (connect=True).

Rules (all conservative: a finding needs an exact tool reference):

config-server-invalid (error): a host-config entry Toolscore cannot use (no command or URL, the deprecated SSE transport, unreadable JSON).
config-unknown-server (warning): an mcp__<server>__<tool> name whose server no host config defines.
config-unknown-tool (error): an mcp__<server>__<tool> name whose server is known and does not expose the tool, or a quoted name written as a tool (`purge_all` tool) that no known server exposes.
config-annotation-contradiction (warning): “use X to delete/create/…” where X declares readOnlyHint: true, or “use X to read/list/…” where X declares destructiveHint: true.

Rules that need a tool inventory are reported as skipped when there is none, never as passed. Claude Code’s built-in tools (Read, Bash, …) and plain prose are never flagged; mcp__plugin_* and mcp__claude_ai_* names come from plugins and connectors outside the repository and are not checked against it.

toolscore.config_lint.CONFIG_RULES = ('config-server-invalid', 'config-unknown-server', 'config-unknown-tool', 'config-annotation-contradiction'): The rule IDs lint_config() reports.

toolscore.config_lint.INVENTORY_RULES = ('config-unknown-tool', 'config-annotation-contradiction'): Rules that need to know which tools each server exposes.

class toolscore.config_lint.ConfigIssue(rule, severity, path, line, message, fix)[source]

One finding of lint_config().

Variables:

rule (str) – The rule ID (see CONFIG_RULES).
severity (str) – "error" or "warning".
path (str) – The file it is in.
line (int | None) – The 1-based line, or None for a whole file.
message (str) – What is wrong.
fix (str) – How to fix it.

Parameters:

rule (str)
severity (str)
path (str)
line (int | None)
message (str)
fix (str)

rule: str

severity: str

path: str

line: int | None

message: str

fix: str

to_json()[source]

A JSON-safe dict of the issue.

Return type:: dict[str, Any]

__init__(rule, severity, path, line, message, fix)

Parameters:

rule (str)
severity (str)
path (str)
line (int | None)
message (str)
fix (str)

Return type:

None

class toolscore.config_lint.ConfigLintReport(instruction_files=list(), host_configs=list(), servers=dict(), inventory=dict(), skipped_rules=dict(), issues=list())[source]

What lint_config() read and found.

Variables:

instruction_files (list[str]) – The instruction files read.
host_configs (list[str]) – The MCP host config files read.
servers (dict[str, str]) – Configured server names, each with the config file defining it.
inventory (dict[str, dict[str, MCPToolDef]]) – For each server with a known tool list, its tools by name.
skipped_rules (dict[str, str]) – Rules that could not run, with the reason.
issues (list[ConfigIssue]) – The findings, in file order.

Parameters:

instruction_files (list[str])
host_configs (list[str])
servers (dict[str, str])
inventory (dict[str, dict[str, MCPToolDef]])
skipped_rules (dict[str, str])
issues (list[ConfigIssue])

instruction_files: list[str]

host_configs: list[str]

servers: dict[str, str]

inventory: dict[str, dict[str, MCPToolDef]]

skipped_rules: dict[str, str]

issues: list[ConfigIssue]

property errors: int: How many error-severity issues there are.

property warnings: int: How many warning-severity issues there are.

to_json()[source]

A JSON-safe dict of the report.

Return type:: dict[str, Any]

__init__(instruction_files=<factory>, host_configs=<factory>, servers=<factory>, inventory=<factory>, skipped_rules=<factory>, issues=<factory>)

Parameters:

instruction_files (list[str])
host_configs (list[str])
servers (dict[str, str])
inventory (dict[str, dict[str, MCPToolDef]])
skipped_rules (dict[str, str])
issues (list[ConfigIssue])

Return type:

None

toolscore.config_lint.discover_files(paths)[source]

Find instruction files and MCP host configs under paths.

Parameters:: paths (list[Path]) – Files or directories (directories are searched recursively, skipping .git, node_modules, virtual environments and build output).
Return type:: tuple[list[Path], list[Path]]
Returns:: (instruction files, host config files), sorted.

toolscore.config_lint.lint_config(paths=None, snapshots=None, connect=False, user_config=None, timeout=30.0)[source]

Lint instruction files and MCP host configs.

Parameters:

paths (list[Path] | None) – Files or directories to read (default: the current directory).
snapshots (list[tuple[str | None, Path]] | None) – Tool inventories as (server name or None, file): a toolscore mcp list --json snapshot or a toolscore mcp pin lock file. Without a name, the snapshot’s own server name is used.
connect (bool) – Start every configured server to list its tools. This runs the commands in the host configs; it is never done by default.
user_config (Path | None) – Also read this user-scope config (~/.claude.json).
timeout (float) – Seconds to wait for each server with connect.

Return type:

ConfigLintReport

Returns:

The report.

Live recorders

See Framework Integration Guide.

Record tool calls live from agent SDKs, as Toolscore traces.

ClaudeAgentSDKRecorder plugs into the Claude Agent SDK’s hooks (PreToolUse, PostToolUse, PostToolUseFailure) and records every tool call with its input, response, error and duration, including MCP tools (mcp__server__tool) and subagent calls.
OpenAIAgentsRecorder is an OpenAI Agents SDK tracing processor: it records every function tool span (function tools and MCP tools) with its input, output and error.

Both keep what they recorded in calls (Toolscore call dicts), take the run’s final answer, and write a trace that every Toolscore command reads (toolscore eval, toolscore honesty). Nothing is sent anywhere; the SDKs stay optional (imported only when a recorder needs them).

class toolscore.recorders.ClaudeAgentSDKRecorder[source]

Record a Claude Agent SDK run through its tool hooks.

Example:

from claude_agent_sdk import ClaudeAgentOptions, query
from toolscore.recorders import ClaudeAgentSDKRecorder

recorder = ClaudeAgentSDKRecorder()
options = ClaudeAgentOptions(hooks=recorder.hooks(), allowed_tools=["Read", "Bash"])
async for message in query(prompt="Run the tests", options=options):
    recorder.observe(message)       # keeps the final ResultMessage text
recorder.save("trace.json")         # then: toolscore honesty trace.json

The hooks return {}: they never block or change a tool call. Merge them with your own hooks with recorder.hooks(existing).

agent = 'claude-agent-sdk'

__init__()[source]

Return type:: None

async pre_tool_use(input_data, tool_use_id, _context)[source]

PreToolUse hook: records the call and its input.

Return type:

dict[str, Any]

Parameters:

input_data (Any)
tool_use_id (str | None)
_context (Any)

async post_tool_use(input_data, tool_use_id, _context)[source]

PostToolUse hook: records the tool’s response and duration.

Return type:

dict[str, Any]

Parameters:

input_data (Any)
tool_use_id (str | None)
_context (Any)

async post_tool_use_failure(input_data, tool_use_id, _context)[source]

PostToolUseFailure hook: records the error.

Return type:

dict[str, Any]

Parameters:

input_data (Any)
tool_use_id (str | None)
_context (Any)

hooks(existing=None)[source]

The hooks to pass as ClaudeAgentOptions(hooks=...).

Parameters:: existing (dict[str, list[Any]] | None) – Your own hooks mapping; the recorder’s matchers are added to it.
Return type:: dict[str, list[Any]]
Returns:: A hooks mapping with a HookMatcher for every tool on PreToolUse, PostToolUse and PostToolUseFailure.

observe(message)[source]

Pass each message query() yields; keeps the final answer and session ID.

Return type:: None
Parameters:: message (Any)

evaluate(expected, **kwargs)

Evaluate the recorded calls (toolscore.evaluate with the final answer).

Parameters:

expected (Any) – The expected calls (as for toolscore.evaluate()), or a contract mapping.
**kwargs (Any) – Passed to toolscore.evaluate().

Return type:

Any

Returns:

The toolscore.EvaluationResult.

save(path, redact=True)

Write the trace as JSON (credentials redacted unless redact=False).

Parameters:

path (str | Path) – The file to write.
redact (bool) – Replace credentials with redacted previews.

Return type:

Path

Returns:

The path written.

to_trace(redact=False)

The recorded run as a Toolscore trace dict.

Parameters:: redact (bool) – Replace credentials the secret scanner recognizes with redacted previews (for a trace you will share or commit).
Return type:: dict[str, Any]
Returns:: {"format": "toolscore", "source", "calls", "final_answer"}.

class toolscore.recorders.OpenAIAgentsRecorder(trace_id=None)[source]

An OpenAI Agents SDK tracing processor that records tool calls.

Example:

from agents import Runner, add_trace_processor
from toolscore.recorders import OpenAIAgentsRecorder

recorder = OpenAIAgentsRecorder()
add_trace_processor(recorder)          # alongside the default exporter
result = await Runner.run(agent, "Book a table for two")
recorder.final_answer = str(result.final_output)
recorder.save("trace.json")

Every function span (function tools, and MCP tools, which the SDK also traces as function spans with mcp_data) becomes a call with its input (parsed JSON arguments), output and error. trace_id restricts recording to one trace; for_trace() reads the calls of one trace.

Parameters:: trace_id (str | None)

agent = 'openai-agents'

__init__(trace_id=None)[source]

Parameters:: trace_id (str | None)
Return type:: None

on_trace_start(trace)[source]

Tracing processor hook (unused).

Return type:: None
Parameters:: trace (Any)

on_trace_end(trace)[source]

Tracing processor hook (unused).

Return type:: None
Parameters:: trace (Any)

on_span_start(span)[source]

Records a tool span’s start (tool name, arguments, time).

Return type:: None
Parameters:: span (Any)

on_span_end(span)[source]

Records a tool span’s output, error and duration.

Return type:: None
Parameters:: span (Any)

shutdown()[source]

Tracing processor hook (unused).

Return type:: None

force_flush()[source]

Tracing processor hook (unused).

Return type:: None

for_trace(trace_id)[source]

The recorded calls of one trace.

Return type:: list[dict[str, Any]]
Parameters:: trace_id (str)

evaluate(expected, **kwargs)

Evaluate the recorded calls (toolscore.evaluate with the final answer).

Parameters:

expected (Any) – The expected calls (as for toolscore.evaluate()), or a contract mapping.
**kwargs (Any) – Passed to toolscore.evaluate().

Return type:

Any

Returns:

The toolscore.EvaluationResult.

save(path, redact=True)

Write the trace as JSON (credentials redacted unless redact=False).

Parameters:

path (str | Path) – The file to write.
redact (bool) – Replace credentials with redacted previews.

Return type:

Path

Returns:

The path written.

to_trace(redact=False)

The recorded run as a Toolscore trace dict.

Parameters:: redact (bool) – Replace credentials the secret scanner recognizes with redacted previews (for a trace you will share or commit).
Return type:: dict[str, Any]
Returns:: {"format": "toolscore", "source", "calls", "final_answer"}.

calls: list[dict[str, Any]]

final_answer: str | None

session_id: str | None

Inspect AI scorer

A Toolscore scorer for Inspect AI (the UK AI Security Institute’s inspect_ai).

toolscore_scorer() scores each sample’s tool calls against what the sample expects, deterministically and without calling a model, and runs the honesty checks on the final answer:

from inspect_ai import Task, task
from inspect_ai.dataset import Sample
from inspect_ai.solver import generate, use_tools
from toolscore.inspect_scorer import toolscore_scorer

@task
def refunds() -> Task:
    return Task(
        dataset=[Sample(
            input="Refund order A-17.",
            metadata={"toolscore": {"required": [{"tool": "issue_refund",
                                                  "args": {"order_id": "A-17"}}]}},
        )],
        solver=[use_tools(issue_refund()), generate()],
        scorer=toolscore_scorer(),
    )

What a sample expects comes from, in order: the scorer’s expected argument, the sample’s metadata["toolscore"] (a list of expected calls or a contract mapping with required / optional / forbidden), or the sample’s target when it is a JSON list or contract.

The score’s value is a dict, one entry per metric:

tool_score: the Toolscore composite score, 0 to 1 (mean);
passed: 1 when the score reaches min_score and no forbidden call was made (accuracy);
false_success: 1 when the honesty checks found an error-severity finding: success claimed after a failure, a refusal overridden with code (mean);
honesty_unknown: 1 when honesty could not be checked (no final answer, or no tool results), so an unknown is never counted as honest (mean).

metadata holds the findings, the tri-state evidence and the calls. Inspect is an optional dependency (pip install inspect-ai); verified with inspect_ai 0.3.287.

toolscore.inspect_scorer.METADATA_KEY = 'toolscore': Sample metadata key holding the expected calls or contract.

toolscore.inspect_scorer.from_inspect_messages(messages)[source]

Tool calls from an Inspect conversation (state.messages).

Each ToolCall on a ChatMessageAssistant is a call (function, arguments, id); the ChatMessageTool with the same tool_call_id gives its result, and its error (a ToolCallError) makes it failed.

Parameters:: messages (Any) – Inspect chat messages (objects, or their model_dump() dicts).
Return type:: list[dict[str, Any]]
Returns:: Toolscore call dicts with tool, args, id and result or error.

toolscore.inspect_scorer.final_answer_from_state(state)[source]

The model’s last answer in an Inspect TaskState (None when it ended on a tool call).

Return type:: str | None
Parameters:: state (Any)

toolscore.inspect_scorer.score_state(state, target=None, expected=None, min_score=0.9, weights=None)[source]

Score one Inspect sample: (values, explanation, metadata) (see the module docstring).

Parameters:

state (Any) – The sample’s TaskState.
target (Any) – The sample’s Target.
expected (Any) – Expected calls or a contract, overriding the sample’s.
min_score (float) – The score a sample needs to pass.
weights (dict[str, float] | None) – Toolscore metric weights.

Return type:

tuple[dict[str, float], str, dict[str, Any]]

Returns:

The score values, an explanation and the metadata.

toolscore.inspect_scorer.toolscore_scorer(expected=None, min_score=0.9, weights=None)[source]

An Inspect AI scorer that checks tool calls and honesty with Toolscore.

Parameters:

expected (Any) – Expected calls or a contract for every sample (default: per sample, from metadata["toolscore"] or a JSON target).
min_score (float) – The Toolscore score a sample needs for passed.
weights (dict[str, float] | None) – Toolscore metric weights (required_call_recall and so on).

Return type:

Any

Returns:

The Inspect scorer.

Framework helpers

toolscore.integrations.claude_agent_sdk_messages(result)[source]

Claude Agent SDK messages (dataclasses or dicts) as Anthropic-style message dicts.

AssistantMessage and UserMessage become {"role", "content"} dicts with typed blocks; a ResultMessage becomes {"type": "result", "result", "is_error", "session_id"}, where the final answer is. Other messages (system, task progress, stream events) are dropped.

Parameters:: result (Any) – The messages query() or ClaudeSDKClient.receive_response() yielded.
Return type:: list[dict[str, Any]]
Returns:: The message dicts, in order.

toolscore.integrations.pydantic_ai_final_answer(result)[source]

The text output of a Pydantic AI AgentRunResult (None for structured output).

Return type:: str | None
Parameters:: result (Any)