A2ABench

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description bears full responsibility. It states the leaderboard is public and ranked by score but omits details like ordering, pagination, side effects, or response format, leaving the agent without critical behavioral context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise (one sentence), which is efficient for a simple tool. However, it could be slightly more structured to improve scannability, such as separating purpose from parameters.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the lack of output schema and only one optional parameter, the description fails to explain the return value or how the parameter affects results. This leaves the agent with significant ambiguity about the tool's output.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, and the description does not mention the 'limit' parameter or its meaning. The agent must infer from the schema alone, which is insufficient for effective use.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action (fetch) and resource (public leaderboard ranked by score). It distinguishes from sibling tools, which deal with questions and submissions, but lacks additional specifics about what the leaderboard contains.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidelines on when to use this tool versus alternatives, nor any mention of exclusion criteria or prerequisites. The description only states what it does, not when it is appropriate.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description must disclose behavioral traits, but it only mentions 'pagination.' It omits details like authentication requirements, rate limits, or whether the list is ordered or filtered.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence of 5 words, front-loading the action and resource. No redundant information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity (one optional param, no output schema), the description is adequate but lacks details about default or max page, question subset, or ordering. It could be more complete with a few extra words.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has 0% description coverage for the 'page' parameter. The description mentions 'pagination' but does not explain the parameter's role, default value, or maximum page number, adding minimal meaning beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('List') and specific resource ('A2ABench questions'), and distinguishes from sibling tools like 'get_leaderboard' and 'submit_benchmark_run'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus alternatives (e.g., 'get_leaderboard' or 'submit_benchmark_run'). The description provides no context, exclusions, or prerequisites.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations are absent, and the description does not disclose side effects, security implications (e.g., api_key handling), or error conditions. The tool is a mutation, but behavior is opaque.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Single sentence is concise but may be overly brief. Important details are omitted, so it does not earn its place fully.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a write operation with 3 required parameters and no output schema, the description lacks essential information about the submission format, scoring process, and potential limitations.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, and the description does not elaborate on parameters. Property names 'entrant_name', 'api_key', and 'submissions' provide minimal hints, but no additional meaning or constraints are added.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description 'Submit benchmark answers for scoring' clearly states the action (submit) and resource (benchmark answers). It distinguishes from siblings 'get_leaderboard' and 'list_benchmark_questions' which are read-only operations.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus alternatives. No prerequisites or context provided, leaving the agent to infer usage from the name alone.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.