Server Quality Checklist

Tool Scores

• set_model_tiersA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description explicitly states it overwrites previously saved tiers entirely, adding behavioral context beyond annotations. Annotations already indicate idempotence, but the description clarifies the overwriting behavior. No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three efficient sentences: first states purpose, second explains process, third details effect. No wasted words, and critical information is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple with one parameter, full schema coverage, and an output schema. The description covers the key behavior (overwrite) and is complete for this complexity level, though it omits error handling details.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and the schema already describes the tiers parameter as a mapping to allowed tier values. The description adds no further meaning, so baseline score is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool saves model tier classifications, using a specific verb (Save) and resource (model tier classifications). It distinguishes from sibling tools like prompt and set_preferences, which deal with prompts and preferences.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use vs. alternatives. The context implies it is for persisting model tiers after sampling/benchmarks, but lacks explicit when-not-to-use or comparison with siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• clear_preferencesA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate destructiveHint=true and idempotentHint=true. The description adds behavioral context by explaining that it reverts to per-call defaults and returns a confirmation string, clarifying the meaning of 'destructive'.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise (two sentences), front-loaded with the purpose, and includes the return type. Every word is necessary.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 0 parameters and annotations cover safety traits, the description is complete: it states what it clears, the effect, and the return value. No gaps for invoking correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0 parameters and 100% schema coverage (vacuously), the description adds no parameter information beyond the schema. Baseline for 0 params is 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Clear' and resource 'persistent preferences', and distinguishes it from siblings like 'set_preferences' by stating it reverts to per-call defaults.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'set_preferences'. It lacks context about prerequisites or appropriate scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• promptA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate destructiveHint=true, but the description adds valuable behavioral details: returns immediately with task ID, client polls, prevents timeouts, and specifies YOLO mode duration. This exceeds annotation-only info.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences with no wasted words. Front-loaded with core action, followed by async behavior and benefit. Highly concise and structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (11 parameters, async, destructive), the description covers the key workflow (async polling, timeout). Output schema exists, so return format is covered elsewhere. Missing minor details like polling mechanism, but sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 91% schema coverage, the description adds minimal parameter detail beyond the schema. It mentions YOLO mode and timeout but does not explain individual parameters further. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it sends a prompt to a CLI runner as a background task, returns immediately with a task ID, and prevents timeouts. This distinguishes it from siblings like batch_prompt (batch) and preference tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains the async nature and timeout prevention, providing clear context for use. However, it does not explicitly contrast with sibling tools like batch_prompt or state when not to use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• batch_promptA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds value beyond annotations by explaining the internal parallel execution mechanism ('asyncio.gather and a semaphore') and that it is a single MCP call. While annotations include destructiveHint=true, the description does not elaborate on destructiveness, but it does not contradict them. The added implementation detail justifies a 4.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, front-loaded with the primary purpose, followed by implementation detail and usage guidance. Every sentence is necessary and non-redundant, achieving maximum efficiency.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the core behavior and differentiation from siblings. An output schema exists, so return format is assumed covered. However, it lacks details on error handling, partial failures, or the implications of destructiveHint, which would enhance completeness for a batch tool. Still, it is largely sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 67%, and the input schema already provides detailed descriptions for all parameters. The tool description itself does not add additional meaning beyond stating the tool sends multiple prompts. With high schema coverage, a baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool sends multiple prompts to CLI runners in parallel, using 'fans out tasks server-side' and explicitly distinguishes it from the sibling 'prompt' tool. The verb 'send' and resource 'multiple prompts' are specific, making the purpose unmistakable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly tells when to use this tool ('primary tool' for multiple prompts) and when to use the sibling 'prompt' ('convenience when sending one task'). It provides clear context and alternatives, leaving no ambiguity.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• set_preferencesA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate idempotency ('idempotentHint': true) and non-destructiveness. The description adds value by explaining persistence across sessions and the update semantics. No contradictions found.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured with three concise paragraphs: purpose, persistence, and example usage. No unnecessary words, every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 24 parameters (none required) and an output schema, the description covers the key behavioral aspects: persistence, update, clearing, and relation to sibling tools. The output schema handles return values, so no further detail needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 58% with descriptions for most fields. The description explains the overall pattern of using 'None' to retain values and 'clear_*' flags to reset, which adds context beyond individual parameter descriptions but does not detail every parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Set'), resource ('persistent preferences'), and scope ('apply to subsequent prompt/batch_prompt calls'). It effectively distinguishes from sibling tool 'clear_preferences' by mentioning its specific function.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit guidance on when to use (set preferences for future calls), persistence across sessions, update behavior, and how to clear fields individually using 'clear_*' flags. Also names 'clear_preferences' as alternative for resetting all fields.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.