Compare two versions of one metric's agreement stats side by side. Requires metric_id, metric_version_a_id, and metric_version_b_id (both versions must belong to that metric). Unavailable for check metrics.
| Name | Required | Description | Default |
|---|---|---|---|
| metric_id | Yes | ||
| metric_version_a_id | Yes | ||
| metric_version_b_id | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden for behavioral transparency. The word 'compare' implies a read-only operation, and the constraint about belonging to the same metric suggests validation behavior. However, the description does not explicitly state whether the tool has side effects, what it returns, or how errors are handled, leaving important behavioral aspects undisclosed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences with the main purpose stated in the first sentence and essential requirements/constraints in the second. Every word earns its place; there is no fluff or redundant information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple comparison tool, the description covers the primary action, key inputs, and an important limitation (check metrics). However, since there is no output schema, the description does not explain what 'agreement stats' are or what the side-by-side output looks like, and it omits error behavior. These gaps make it only partially complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has three required integer parameters with 0% schema description coverage, so the description must add semantic meaning. It names all three parameters and, crucially, adds the relationship constraint that both version IDs must belong to the given metric_id. This goes beyond the bare schema and helps the agent understand the domain logic, although it doesn't detail value formats or enums.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description 'Compare two versions of one metric's agreement stats side by side' clearly identifies the specific verb (compare), the resource (metric's agreement stats), and the scope (two versions of one metric). This distinguishes it from sibling tools like metrics_get or metric_versions_list, which do not directly offer side-by-side version comparison.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly states the required input parameters and a key constraint ('both versions must belong to that metric'), which clarifies when the tool is applicable. It also provides an exclusion ('Unavailable for check metrics'). However, it does not name any alternative tools for cases where the criteria aren't met, so it earns a 4 rather than a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Add one secure layer between your agents and this server.
Each tool targets a distinct resource and action, with clear separation across agreements, datasets, judges, metrics, prompts, runs, tags, and usage. Even similar tools like datasets_create vs datasets_create_from_url and runs_generate vs runs_rerun are explicitly differentiated in their descriptions.
The overwhelming majority of tools follow a consistent plural_resource_action snake_case pattern (e.g., datasets_create, metrics_update, runs_retry_failures). The only slight deviation is promptfoo_import, but it is still descriptive and does not break the overall predictability.
With 54 tools, the server far exceeds the 25+ threshold considered too many, and approaches the 50+ extreme mismatch level. Even for a broad LLM evaluation platform, this count is excessive and likely to overwhelm agents, making tool selection more error-prone.
The toolset provides full CRUD for core resources (datasets, metrics, prompts, runs, tags) plus lifecycle operations like publish, generate, regrade, and retry. It also includes cross-cutting utilities (usage, import, provider credentials). Minor gaps exist, such as no update/delete for agreements and no cross-run response search, but these are non-essential for the primary workflows.
