Server Quality Checklist

Tool Scores

• web_fetchA
  3.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations present, the description carries the transparency burden. It discloses the HTTP method and the crucial 'offline stub by default' behavior, which is valuable caveat. However, it does not detail response handling, error behavior, or potential side effects, leaving some gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, front-loaded with the primary action and resource. Every word adds value, including the important 'offline stub' caveat. It is appropriately concise with no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter tool with an output schema, the description covers the essential purpose and a key behavioral trait. However, it lacks usage guidance and alternative differentiation, and the absence of annotations increases the need for more contextual details about expected behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides no description for the 'url' parameter (coverage 0%), so the description must compensate. It only says 'Fetch a URL,' which essentially restates the parameter name and adds no information about URL format, encoding, or expected constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a clear action ('Fetch a URL') with a specified resource and method ('over HTTP GET'), which distinguishes it from sibling tools like calculator and file_read. The additional caveat 'offline stub by default' adds useful specificity about the tool's behavior.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no guidance on when to use this tool versus alternatives, no prerequisites, and no exclusions or limitations beyond the stub note. It only implies usage for fetching URLs, but does not offer explicit contextual direction.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• calculatorA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of behavioral disclosure. It discloses supported operators and parentheses, but does not mention error behavior (e.g., division by zero), precision, or whether the operation is side-effect-free. The word 'pure' hints at a pure function but is not explicit.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single, front-loaded sentence that immediately states the tool's purpose and then provides necessary syntax details. No redundant or filler content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple calculator tool with a single string parameter and an output schema, this description is sufficiently complete. It defines the input format and scope, and the output schema covers return values, so no additional behavior needs to be explained.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema only defines 'expression' as a string, with 0% coverage of its format. The description compensates by specifying the allowed operators (+ - * / // % **) and parentheses, giving the agent a clear understanding of the expected input syntax.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Evaluate') and a resource ('pure-arithmetic expression'), and it distinguishes itself from sibling tools (file_read, web_fetch) by clearly indicating a math evaluation function. The operator list further specifies the scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It clearly implies when to use it (for arithmetic calculations) and the 'pure-arithmetic' qualifier excludes non-math uses. However, it lacks explicit 'when not to use' or named alternatives, though the sibling tools are obviously different in purpose.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• file_readA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are present, so the description carries the full burden. It explicitly states this is a read operation (non-destructive) and restricts use to UTF-8 text files inside the sandbox, which is meaningful disclosure. It lacks details on error handling or path traversal, but these are less critical for a simple read tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no filler. Every word contributes meaning, making it highly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple, and the description covers the essential aspects: purpose, file type, and scope. An output schema exists, so return values are covered. It doesn't mention error cases, but for a basic read tool this is adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, but the description compensates by specifying that the path refers to a UTF-8 text file inside the sandbox. This provides crucial context for the 'path' parameter, making its meaning clear beyond just a bare string.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb ('Read') and resource ('UTF-8 text file') with a clear scope ('inside the sandbox directory'). It distinguishes the tool from siblings (calculator, web_fetch) by focusing on local file reading.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context that this tool reads files within the sandbox directory, implying it is not for external resources. However, it does not explicitly mention alternatives or when not to use it, so it falls short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.