Server Quality Checklist

Tool Scores

• codebase_cancel_jobA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry the full burden. It mentions cancellation but fails to disclose important behavioral aspects such as irreversibility, scope of effect, or error states (e.g., what happens if the job is already complete or not found).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single, front-loaded sentence with no wasted words. It efficiently conveys the purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of a cancellation action and the presence of an output schema (not shown), the description might be minimally sufficient. However, it lacks details on error handling or state dependencies, leaving gaps for an agent to infer.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% for the single parameter, and the description in the schema already explains the source of job_id. The tool description adds minimal value beyond stating 'queued or running', which is not critical for parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Request cancellation' and identifies the resource as 'a queued or running analysis job'. It distinguishes from siblings which are about getting results, status, or starting jobs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use (to cancel jobs) but does not explicitly state when not to use or mention alternatives. The context of siblings gives some guidance, but not explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codebase_list_librariesA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It discloses a read operation, but lacks details on completeness, performance, or any limitations. Adequate but minimal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence with no extraneous information. It is appropriately sized and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (no parameters, no annotations, clear output), the description is sufficient to inform an agent. No additional details are necessary.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With zero parameters and full schema coverage, the description adds little beyond stating the return value. Baseline is 4, which is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Return' and the resource 'public names of local libraries/codebases'. It distinguishes from sibling tools that focus on job management, making the tool's purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for listing available libraries, but does not explicitly state when to use this tool versus alternatives. No guidance on prerequisites or context is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codebase_start_job_analysisA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses async behavior and return content (analysis, recommendations, etc.), but lacks detail on error handling, permissions, or side effects. With no annotations, more behavioral context is expected.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured with bold emphasis, front-loading purpose. Some repetition of instructions between description and parameter schema, but overall efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 2 parameters, output schema, and siblings, description covers async nature, single-entity focus, sibling precedence, and usage pattern. Missing error/cancellation notes but still solid.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema covers 100% of parameters; description adds value by reinforcing the query's required focus and one-topic rule. Adds meaningful guidance beyond schema basics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States 'Start async codebase analysis' with specific verbs and resource, and distinguishes from siblings by emphasizing single-entity analysis and preference for related_files_search. Clear and differentiated.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly suggests using codebase_start_job_related_files_search first, and mandates one topic per request with wait-for-result pattern. Provides clear when-to-use and when-not-to.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codebase_start_job_related_files_searchA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses the async nature, the requirement to wait for results, and the single-topic constraint. The presence of an output schema reduces the need to explain return values. Minor gap: no mention of error cases or edge conditions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the main action and includes necessary usage instructions. Some redundancy in the bolded repetition, but overall efficient for the information conveyed.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (async, required parameters) and the presence of an output schema, the description covers the purpose, usage pattern, and relationship to sibling tools adequately. Could add a note on expected result content, but not critical.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters. The description adds emphasis on usage ('one topic per request') but does not provide new semantic meaning beyond what the schema already offers.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Start async related-file discovery') and resource ('find files relevant to single entity/action before codebase_start_job_analysis'), distinguishing from sibling tools by specifying its sequential usage and one-topic-per-request constraint.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly provides usage guidelines: 'One topic or one context per request → wait for the result → send the next request!' and contrasts with trying to find multiple things at once, guiding the agent on correct invocation patterns.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codebase_get_job_resultA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses significant behavioral traits: internal wait up to 50s, dependency on prior status polling, and a rule against finalizing during polling. However, there is minor ambiguity: mentioning 'latest partial output' suggests it may be called before terminal status, while the instruction to poll until success/failure implies it should not be. This slight inconsistency prevents a perfect score.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is four sentences, each earning its place: purpose first, then crucial usage rules and warnings. It is well-structured, front-loaded, and free of fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a complex tool with no annotations, the description covers the essential context: polling sequence, wait behavior, failure handling, and a hard rule about not finalizing responses. The minor ambiguity about partial output versus terminal polling keeps it from being fully complete, but it is otherwise thorough.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The only parameter, job_id, has 100% schema description coverage and is fully explained in the schema as the id returned by the start job tools. The description adds no extra semantics beyond the schema, so baseline 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb+resource ('Return the final result, error, or latest partial output for an analysis job') that clearly distinguishes it from sibling tools like codebase_get_job_status (status) and codebase_start_job_* (starting jobs). It unambiguously states what the tool does.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit when-to-use guidance: poll codebase_get_job_status until success/failure, then use this tool. It warns about the 50s internal wait, cautions against assuming failure before terminal status, and forbids finishing the response while polling is running. This is a complete workflow context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codebase_get_job_statusA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full responsibility. It discloses that the tool internally waits up to 50s per request, warns against premature failure assumptions, and asserts a strict rule about not finishing responses during polling. This goes well beyond a generic 'get status' description.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with a bold, clear purpose statement, followed by compact actionable instructions. Every sentence adds value—polling requirement, 50s wait, terminal status, and the prohibition on premature response—with no redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter status polling tool with an output schema, the description fully covers purpose, usage, and behavioral constraints. It explains the relationship to the sibling job-starting tools and defines the polling loop, making it complete for agent invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides 100% coverage for the single parameter (job_id), including its source and type. The description does not add further parameter-specific detail, so the baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Get async analysis job status' and specifically names both job-starting tools (`codebase_start_job_related_files_search` and `codebase_start_job_analysis`), making the tool's purpose unambiguous and distinguishing it from siblings that start jobs or get results.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Usage guidance is explicit: 'You must poll using `codebase_get_job_status` tool until `success`/`failure`' and 'Avoid assuming failure before terminal status.' It also forbids finishing responses while polling, clearly defining when to use the tool and how to handle non-terminal states.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.