Server Quality Checklist

Tool Scores

• start_debug_loopC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden for behavioral disclosure. It mentions the tool returns an instruction for TDD tasks and that the PR body includes a diagnosis writeup, giving some insight into outputs and workflow. However, it lacks critical details like whether this is a read-only or mutating operation, error handling, or any side effects, which are essential for a tool that starts processes.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately concise with two sentences that front-load the core action and follow with output details. Each sentence adds value: the first defines the tool's purpose, and the second explains the return behavior and PR inclusion. There's no wasted text, though it could be slightly more structured for clarity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (starting a debug process with no annotations or output schema), the description is incomplete. It covers the basic action and some output hints but misses key behavioral aspects like mutation status, error cases, or interaction with sibling tools. Without annotations or an output schema, more detail is needed to fully inform the agent about this process-initiating tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, with the 'symptom' parameter fully documented as a natural-language bug description. The description adds no additional parameter semantics beyond what the schema provides, such as format examples or constraints. This meets the baseline score of 3, as the schema does the heavy lifting without extra value from the description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Start a debug loop from a symptom description.' It specifies the verb ('Start'), resource ('debug loop'), and input trigger ('symptom description'). However, it doesn't explicitly differentiate from sibling tools like 'start_loop' or 'advance_loop', which reduces clarity about when to choose this specific debug variant.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides minimal guidance on when to use this tool. It mentions the tool initiates a debug loop from a symptom, but offers no explicit when/when-not criteria or alternatives. While it implies usage for bug diagnosis, it doesn't compare to sibling tools like 'start_loop' or specify prerequisites, leaving the agent with little contextual direction.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• loop_statusB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It indicates this is a read operation ('Get'), but doesn't specify whether it requires authentication, has rate limits, returns real-time or cached data, or handles errors. The description adds minimal behavioral context beyond the basic action, missing key details for safe and effective use.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core action ('Get the current status of the loop') and lists the retrieved data points without unnecessary words. Every element serves a purpose, making it highly concise and well-structured for quick understanding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (0 parameters, no output schema, no annotations), the description is adequate but has gaps. It covers what data is retrieved, but without annotations or an output schema, it doesn't explain the return format (e.g., structure of the status object) or behavioral aspects like error handling. For a status-checking tool, this leaves some contextual needs unmet.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0 parameters with 100% coverage, so no parameter documentation is needed. The description appropriately adds no parameter information, focusing instead on the tool's purpose and output semantics. This aligns with the baseline expectation for tools without parameters, as the schema fully covers the input structure.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with the verb 'Get' and specifies the resource 'current status of the loop', including what information is retrieved (phase, branch, task list, failure reason, PR URL). It distinguishes itself from sibling tools like 'advance_loop', 'start_debug_loop', and 'start_loop' by focusing on status retrieval rather than initiation or progression, though it doesn't explicitly name these alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage by listing the specific data points retrieved (e.g., phase, failure reason), suggesting it's for monitoring or checking loop progress. However, it lacks explicit guidance on when to use this tool versus alternatives (e.g., when to check status vs. start or advance a loop) or any prerequisites, leaving usage context inferred rather than stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• start_loopA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses the tool's purpose and procedural flow (phases, interaction with advance_loop), but lacks details about potential side effects, error conditions, authentication needs, or what 'DONE or FAILED' states entail. It provides basic behavioral context but misses important operational details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficiently structured in two sentences: the first states the purpose and return value, the second provides crucial procedural guidance and lists all phases. Every element serves a clear purpose with zero wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (managing a multi-phase development loop) and the absence of both annotations and output schema, the description provides adequate procedural context but lacks details about return values, error handling, and operational constraints. It's complete enough to understand the basic workflow but leaves important implementation questions unanswered.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all three parameters thoroughly. The description adds no additional parameter information beyond what's in the schema. According to scoring rules, when schema coverage is high (>80%), the baseline is 3 even with no param info in the description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Start a new TDD development loop') and the resource (the loop itself), with a specific verb. However, it doesn't explicitly distinguish this from its sibling 'start_debug_loop' beyond the name difference, leaving some ambiguity about when to choose one over the other.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage guidance: it tells the agent to 'Keep calling advance_loop after each step until the loop reaches DONE or FAILED' and lists all the phases of the loop. This gives clear procedural context for when and how to use this tool in relation to its sibling 'advance_loop'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• advance_loopA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions the loop persists state between calls, which is useful context. However, it doesn't describe error handling, rate limits, authentication needs, or what happens if called out of sequence. For a stateful tool with no annotations, more behavioral details would be helpful.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise (two sentences) and front-loaded with the core purpose. Every word earns its place, with no redundant information. The structure moves from primary function to usage guidance efficiently.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (stateful loop advancement with multiple event types) and 100% schema coverage but no output schema or annotations, the description is mostly complete. It explains the core workflow and when to use it, though it could benefit from mentioning what the tool returns (the 'next instruction') more explicitly since there's no output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all 6 parameters thoroughly with their purposes and event-specific usage. The description doesn't add any parameter-specific information beyond what's in the schema, making the baseline score of 3 appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('Report the outcome', 'get the next one') and distinguishes it from siblings by specifying it's for after completing each phase step in a loop. It explicitly mentions the loop persists state, which differentiates it from start_loop or loop_status.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage guidance: 'Call this after completing each phase step.' This tells the agent precisely when to use this tool versus alternatives like start_loop (for initiation) or loop_status (for checking status without advancing).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.