agent-eval

Overview Schema Source Related Servers Discussions Admin

run

Compare two pre-computed score files to get a statistically grounded verdict on whether an LLM agent's performance change is a real regression, improvement, or just run-to-run noise.

Instructions

Run the agent-regress CLI as a subprocess and return its parsed --json output, so an agent can get a statistically grounded verdict on whether an LLM agent's behavior actually shifted between two versions instead of eyeballing two numbers.

Call this after you already have two pre-computed sets of per-run scores (one JSON array of numbers per agent version, produced by whatever harness ran each version) and you need to know whether the difference between them is a real regression/improvement or just run-to-run noise. Do not call it to generate those scores in the first place -- it only compares numbers that already exist on disk. Right now the CLI has one subcommand, compare; pass ["--help"] or ["compare", "--help"] as args to discover any subcommands/flags added since this description was written.

Behavior: this is a read-only, idempotent operation with no network calls -- it spawns a local subprocess, reads the two score files you point it at, and writes nothing to disk itself. No API keys or network access are required, but the paths passed via --version-a-results/--version-b-results must already exist and contain valid JSON arrays of numbers. On a normal run (including a REGRESSED verdict with --fail-on-regression, which legitimately exits 1) the tool returns the CLI's parsed JSON report. On an actual failure (bad args, CLI not on PATH, unparseable output) it instead returns a dict with an "error" key and, where available, the captured "stderr"/"stdout".

Parameters: args: list[str] -- CLI argv, NOT a shell string. Include the subcommand and its flags exactly as agent-regress would take them on the command line; --json is appended automatically if you omit it. Examples: ["compare", "--version-a-results", "a.json", "--version-b-results", "b.json"] ["compare", "--version-a-results", "a.json", "--version-b-results", "b.json", "--metric", "tool_accuracy", "--min-effect", "0.3", "--fail-on-regression"] ["compare", "--help"]

Returns (JSON object) on success: metric, verdict (one of REGRESSED / STABLE / IMPROVED / INSUFFICIENT_DATA), p_value, effect_size, ci_lower, ci_upper, n_a, n_b, mean_a, mean_b, std_a, std_b, mean_delta, p_threshold, min_effect, and warnings (e.g. low sample size). On failure: error (and stderr/stdout when captured).

Input Schema

Name	Required	Description	Default
`args`	Yes

Output Schema

Name	Required	Description	Default
No arguments

Schema Changelog

Changes observed during successful MCP inspections.

First observedAug 10, 2026

TDQS

A5/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It discloses read-only/idempotent behavior, no network calls, no disk writes, no API keys/network requirement, path prerequisites, normal success return shape, and the exact failure-mode return structure, including the exit-1 REGRESSED case.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is long but densely informative; every sentence adds operational value, from invocation conditions to examples to failure handling. The most important scoping information (what it does and when to call it) is front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a one-parameter CLI wrapper with no annotations, the description is complete: it covers prerequisites, expected return fields, error behavior, examples, and a discovery path for future flags. Nothing needed to invoke it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema only says args is an array of strings, so it provides almost no semantic weight. The description compensates fully by saying it is CLI argv, not a shell string, that --json is appended automatically, and by giving three concrete, realistic examples with flags.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names the exact operation ('Run the agent-regress CLI as a subprocess and return its parsed --json output') and explains the purpose: getting a statistically grounded verdict on whether an agent's behavior changed. It also explicitly differentiates the tool from score-generation tasks, so an agent knows what it is and is not.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It states when to call it: after two pre-computed sets of per-run scores exist and a regression/improvement verdict is needed. It explicitly says not to call it to generate those scores, and it advises using --help to discover newly added subcommands/flags, which is strong routing guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools

runA