Server Quality Checklist

Tool Scores

• roll_diceC
  2.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure but only states the basic action. It doesn't describe what the tool returns (e.g., results format, randomness details), error conditions, or any behavioral traits like side effects or limitations, leaving significant gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with no wasted words, making it appropriately concise. However, it's under-specified rather than optimally structured, as it could benefit from slightly more detail without losing efficiency.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of a dice-rolling tool with 2 parameters, 0% schema coverage, no annotations, and no output schema, the description is incomplete. It doesn't explain the return values, parameter details, or behavioral context, making it inadequate for an agent to use effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate but adds minimal meaning. It mentions 'notation' without explaining what it is (e.g., dice notation like '2d6'), and doesn't address 'num_rolls' at all. This fails to compensate for the lack of schema documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Roll the dice with the given notation' clearly states the action (roll) and resource (dice), but is vague about what 'notation' entails or how it differs from sibling tools like repair_cost and web_search. It doesn't specify the format or examples of dice notation, leaving the purpose somewhat ambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives or in what context it's appropriate. The description lacks any mention of prerequisites, typical use cases, or comparisons to sibling tools, offering no help for an agent deciding when to invoke it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• repair_costC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden for behavioral disclosure. While 'Get' implies a read operation, the description doesn't disclose important behavioral traits like whether this is an estimate vs. actual cost, data source, accuracy limitations, rate limits, authentication needs, or what happens when invalid inputs are provided. This is inadequate for a tool with no annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero waste. It's appropriately sized for a simple tool and front-loads the core purpose immediately. Every word earns its place in communicating the essential function.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (2 required parameters, no annotations, no output schema), the description is incomplete. It doesn't explain what the tool returns, how estimates are calculated, error conditions, or parameter requirements. For a tool with no structured metadata, the description should provide more contextual information to guide proper usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description must compensate by explaining parameter semantics, but it provides no information about the two required parameters (repair_type, zip_code). It doesn't explain what values are expected, formats, constraints, or examples. The description adds no value beyond what the bare schema provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('Get') and resource ('repair cost estimate for home repairs'), making it immediately understandable. It doesn't distinguish from siblings (roll_dice, web_search), but those are unrelated tools, so differentiation isn't needed here.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites, limitations, or scenarios where this tool is appropriate versus other approaches. With no annotations to provide context, this leaves the agent with insufficient usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• web_searchC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It mentions 'Search the web' but doesn't disclose behavioral traits like rate limits, authentication needs, result format, pagination, or whether it's read-only or has side effects. The description is minimal and lacks essential operational context for an AI agent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with no wasted words. It's appropriately sized for a simple tool and front-loaded with the core action. Every part of the sentence contributes to understanding the tool's basic function.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (web search can involve nuances like result types, sources, and limitations), no annotations, no output schema, and low parameter semantics, the description is incomplete. It doesn't address what the tool returns, how results are structured, or any operational constraints, making it inadequate for effective use by an AI agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds minimal meaning beyond the input schema. It mentions 'query' as the parameter but doesn't elaborate on syntax, format, or examples. With schema description coverage at 0% and 1 parameter, the baseline is 4 for 0 parameters, but here it's 1 parameter with no schema descriptions, so the description partially compensates by implying the parameter's purpose but remains vague.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Search') and resource ('the web') with a specific purpose ('for information about the given query'). It distinguishes itself from sibling tools like 'repair_cost' and 'roll_dice' by focusing on web information retrieval. However, it doesn't specify what type of web search (e.g., general, news, images) or differentiate from potential similar tools not present in this server.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention any prerequisites, limitations, or contexts where this tool is preferred over other methods. With sibling tools being unrelated ('repair_cost', 'roll_dice'), there's no explicit comparison or exclusion criteria provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.