Server Quality Checklist

Tool Scores

• delete_programC
  2.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description must disclose behavioral traits, but it only mentions deletion of program and traces. It omits irreversibility, authorization needs, or cascading effects, which are critical for a destructive tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, making it concise, but it lacks structure and important details, bordering on under-specification.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema and minimal annotations, the description is incomplete. It provides basic purpose but lacks usage context, behavioral detail, and parameter guidance.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description adds no meaning to the single parameter 'program_id' beyond its existence. The parameter is self-explanatory, but the description fails to compensate for the lack of schema documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Delete' and the resource 'saved program and all its traces,' distinguishing it from siblings like get_program and run_program.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool vs alternatives or any exclusions. The description lacks context on prerequisites or when deletion is appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_programB
  3.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description implies a read operation ('Retrieve') but does not disclose any behavioral traits such as error handling or permissions; with no annotations, the burden is on the description, yet it remains minimal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence that front-loads the action and resource, but its brevity sacrifices valuable context like parameter details and usage guidance.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple retrieval tool with no output schema, the description minimally covers the return type ('Program JSON'), but lacks details on edge cases or behavior, making it adequate but not thorough.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds no meaning to the program_id parameter beyond its name; with 0% schema description coverage, the description fails to compensate, leaving the parameter's semantics entirely implicit.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves a saved Program JSON by program_id, using a specific verb and resource, which distinguishes it from sibling tools like delete_program and list_programs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives like get_trace or list_programs is provided, leaving the agent without context for selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_traceA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description must disclose behavioral traits. It indicates a read operation but fails to mention what happens for invalid trace_ids, authentication needs, or limitations. The brevity leaves significant gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single sentence of 12 words, perfectly concise and front-loaded. Every word earns its place with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  No output schema exists, so the description should explain return format. 'Full Trace JSON' is vague but adequate. Missing error details and more specific return info make it minimally complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with trace_id described as 'UUID returned by run_program.' The description adds context ('completed run'), which is helpful but minimal. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves the full Trace JSON for a completed run by trace_id. The verb 'Retrieve' and resource 'Trace JSON' are specific, and the tool is distinctly different from siblings like run_program, get_program, etc.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use after a completed run and mentions trace_id from run_program, but it lacks explicit guidance on when to use versus alternatives, such as not using for running programs or error handling.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_programA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description partially discloses behavior: it performs execution, returns specific fields, and optionally saves. However, it omits details like side effects (other than save), permissions, idempotency, or potential errors.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, focused sentence that communicates core purpose and return fields. It is clear and front-loaded, though the return list could be better integrated (e.g., 'returns a result with...').

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description lists return fields but lacks details on their structure or format. For an execution tool with nested input, it provides sufficient high-level info but could be more complete regarding error cases and complex behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% (both parameters described). The tool description adds no new information beyond the schema’s own descriptions (e.g., 'nano_vm.Program JSON' and 'Optional name to save'). Baselines at 3 with no extra value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool executes a nano-vm Program and returns trace_id, status, step count, and cost. It distinguishes from sibling tools (delete, get, list, get_trace) by focusing on execution and optional persistence.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use vs alternatives or prerequisites. While sibling tools imply usage for execution, the description does not provide context for when not to use or any required setup.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_programsA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Since no annotations exist, the description bears full burden. It correctly indicates a read-only operation ('List') but does not disclose any pagination, limitation, or ordering behavior. For a zero-parameter tool, basic transparency is adequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single, front-loaded sentence that conveys the action, resource, and return fields without any extraneous words. Every word is necessary.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list tool with no output schema, the description fully explains what the tool returns (id, name, created_at). No additional context is needed given the tool's simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% (no parameters), so baseline is 3. Description adds no parameter information, which is acceptable as there are none to document.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'List' and resource 'all saved programs', explicitly stating the fields returned (id, name, created_at). It clearly distinguishes from sibling tools like get_program (single) and run_program (execution).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use list_programs versus get_program or other siblings, nor any hints about prerequisites or alternatives. The description is purely functional.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.