agent-eval
Server Configuration
Describes the environment variables required to run the server.
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||
Instructions
Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.
This server publishes no instructions, or was last inspected before Glama recorded them.
Capabilities
Features and capabilities supported by this server
Protocol revision2025-11-25
| Capability | Details |
|---|---|
| tools | {
"listChanged": false
} |
| prompts | {
"listChanged": false
} |
| resources | {
"subscribe": false,
"listChanged": false
} |
| experimental | {} |
Tools
Functions exposed to the LLM to take actions
| Name | Description |
|---|---|
| runA | Run the agent-regress CLI as a subprocess and return its parsed --json output, so an agent can get a statistically grounded verdict on whether an LLM agent's behavior actually shifted between two versions instead of eyeballing two numbers. Call this after you already have two pre-computed sets of per-run scores (one JSON array of numbers per agent version, produced by whatever harness ran each version) and you need to know whether the difference between them is a real regression/improvement or just run-to-run noise. Do not call it to generate those scores in the first place -- it only compares numbers that already exist on disk. Right now the CLI has one subcommand, Behavior: this is a read-only, idempotent operation with no network calls -- it spawns a local subprocess, reads the two score files you point it at, and writes nothing to disk itself. No API keys or network access are required, but the paths passed via --version-a-results/--version-b-results must already exist and contain valid JSON arrays of numbers. On a normal run (including a REGRESSED verdict with --fail-on-regression, which legitimately exits 1) the tool returns the CLI's parsed JSON report. On an actual failure (bad args, CLI not on PATH, unparseable output) it instead returns a dict with an "error" key and, where available, the captured "stderr"/"stdout". Parameters:
args: list[str] -- CLI argv, NOT a shell string. Include the subcommand and its flags exactly as Returns (JSON object) on success: metric, verdict (one of REGRESSED / STABLE / IMPROVED / INSUFFICIENT_DATA), p_value, effect_size, ci_lower, ci_upper, n_a, n_b, mean_a, mean_b, std_a, std_b, mean_delta, p_threshold, min_effect, and warnings (e.g. low sample size). On failure: error (and stderr/stdout when captured). |
Prompts
Interactive templates invoked by user choice
| Name | Description |
|---|---|
No prompts | |
Resources
Contextual data attached and managed by the client
| Name | Description |
|---|---|
No resources | |
TDQS
Scored across 1 tool
Only one tool exists, so there is no possibility of ambiguity or overlap. The single tool has a clear, well-defined purpose (running a CLI comparison).
With only one tool, the naming is internally consistent, but the name 'run' is generic and does not convey the tool's statistical comparison purpose. It follows no distinctive pattern, but there are no deviations to penalize heavily.
A single tool is on the thin side for a server, even one with a narrow purpose. While the tool fully addresses the comparison task, the lack of any supplementary tools (e.g., for generating scores or listing subcommands) makes the server feel minimal.
The tool covers the core need—comparing two sets of scores and returning a statistical verdict. The description explicitly scopes it to comparison only, and within that scope it is complete. However, it relies on the CLI to potentially expose other subcommands (via --help), so there could be untapped functionality, leaving a minor gap.