ai-eval

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description must disclose behavior, but it only says 'compare and rank' without explaining ranking criteria, output format, or side effects. It doesn't clarify whether rankings are returned as scores, an ordered list, or something else.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence and easy to read, but it is under-specified rather than properly concise. It front-loads the core action but omits necessary details at the cost of usefulness.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, no annotations, and only a one-sentence description, the tool is insufficiently specified for an agent to understand return values, ranking criteria, or when to invoke it correctly. The description covers the basic purpose but lacks essential operational context.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, and the description does not elaborate on the prompt or responses parameters. The property names are self-explanatory, but the description adds no meaning beyond the schema, such as constraints or relationship between parameters.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: comparing and ranking multiple AI responses. The verb 'compare and rank' is specific, and 'multiple AI responses' distinguishes it from score_response, though it doesn't explicitly mention sibling alternatives.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided on when to use this tool versus score_response or text_metrics. The description does not mention alternatives, prerequisites, or typical use cases, leaving the agent to infer context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the transparency burden. It adds some context by noting 'heuristic metrics' (length, relevance, structure, completeness), which implies non-deterministic or approximate scoring. However, it does not disclose the output format, edge cases, or any limitations beyond listing metrics.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence that leads with the primary action and resource, then lists the metrics. Every word adds value, with no redundancy or filler. It is appropriately sized for the tool's simplicity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has three parameters, no output schema, and no annotations, so the description must explain return values and usage context. It fails to state what the score looks like (numeric range, object, etc.) or how to interpret the result. This leaves a significant gap for the agent to invoke it confidently.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The description implicitly references 'prompt' and 'response' but adds no explicit parameter-level detail beyond the schema. The optional 'criteria' parameter is not mentioned in the description, but the schema already documents it.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb ('Score'), a clear resource ('an AI response against a prompt'), and lists the heuristic metrics used. It distinguishes from sibling tools like 'compare_responses' and 'text_metrics' by emphasizing prompt-relative evaluation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description clearly indicates when to use the tool (whenever an AI response needs scoring against a prompt). However, it does not explicitly mention alternatives or situations where a sibling tool would be more appropriate, lacking exclusions or when-not-to-use guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden for behavioral disclosure. It explicitly states the metrics returned, which is transparent, but it does not explain how 'estimated tokens' are estimated or what formula/scales are used for 'readability grade'. It also doesn't mention any side effects, though the tool appears read-only.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, front-loaded sentence that lists the key metrics. It is succinct and avoids unnecessary filler, every word earning its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool is simple, and the description covers the core purpose, but because there is no output schema, the return format is undefined. The description lists metrics but doesn't specify the structure (e.g., JSON object keys, value types) or clarify the nature of 'estimated tokens' and 'readability grade', which are common sources of ambiguity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has a single 'text' parameter with 0% coverage, so the description must compensate. The description adds meaning by indicating that the input should be text with sentences and words, which is inferred from the mentioned metrics like sentence count. However, it doesn't specify format constraints (e.g., plain text vs markdown) or maximum length.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Get') and the resource ('text quality metrics'), then lists specific metric types: word count, sentence count, estimated tokens, readability grade. This distinguishes it from sibling tools like compare_responses and score_response, which focus on comparison and scoring.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage when text metrics are needed but provides no explicit guidance on when to use this tool versus compare_responses or score_response. No exclusions or alternatives are mentioned, leaving the agent to infer the appropriate context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.