Benchmark recorded judges on synthetic gold labels to evaluate agreement; default runs an offline demo.
Benchmark recorded judges on synthetic gold labels; default is an offline demo.
| Name | Required | Description | Default |
|---|---|---|---|
| n_bins | No | ||
| fixture | No | Synthetic v1 fixture; omit for bundled demo. At most 1000 items and 16 judges. |
Changes observed during successful MCP inspections.
v0.1.0Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly, idempotent, non-destructive, closed-world behavior, so the safety profile is covered. The description adds a real behavioral caveat not present in structured data — that scoring runs against synthetic gold labels and defaults to an offline demo — but says nothing about runtime, cost, or what a benchmark run actually produces.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with a semicolon separating the core action from the default-behavior note. No filler, no repetition of the title or schema.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a read-only tool with no output schema the bar is lower, but with one of two parameters undocumented, a nested fixture object, and no explanation of what a benchmark returns or how it relates to the other jevlab_* tools, the description is thinner than the tool's surface area warrants.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 50%: the fixture parameter is documented in the schema, but n_bins has no description anywhere and its meaning (histogram binning? something else?) is left entirely to inference. The description text itself contributes no parameter information at all.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific verb and resource ('benchmark recorded judges') plus the data source ('synthetic gold labels'), so an agent knows this evaluates judges rather than auditing or calibrating them. It does not explicitly differentiate itself from jevlab_calibrate or jevlab_bias_audit, and 'recorded judges' is jargon that isn't defined.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The clause 'default is an offline demo' hints at a zero-argument invocation mode but gives no when-to-use guidance, no conditions for supplying a fixture, and no routing against the sibling tools. The agent must infer when to prefer this over calibrate or bias_audit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.