Server Quality Checklist

Tool Scores

• search_symbolC
  2.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must disclose key behaviors. It only mentions ranking without explaining what that entails (e.g., algorithmic details, side effects, or result structure). The behavior is opaque.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, which is concise, but it lacks structure and does not front-load critical information. Every word earns its place, but the overall information density is low.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has three parameters and no annotations, the description is insufficient. It does not explain the output (despite an output schema existing) or the ranking mechanism, leaving significant gaps for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, yet the description adds no information about the parameters (repository_id, query, limit). The agent cannot infer what values are expected or how they affect the search.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Search' and the resource 'symbols', adding a qualifier about lexical and code-structure ranking. This distinguishes it from sibling tools like read_file and find_references, though it does not explicitly contrast them.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. There is no mention of prerequisites, context, or exclusions, leaving the agent without decision support.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• read_fileC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must disclose behavioral traits. It only mentions 'bounded line range', but omits critical details such as error handling, performance implications, file size limits, or what happens if line range exceeds file length. This is insufficient for safe and effective tool usage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence with no unnecessary words. It is front-loaded with the key action. However, its brevity sacrifices important details, making it borderline under-specified for a tool with multiple parameters.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given four parameters, no annotations, and an output schema, the description is too sparse. It does not explain the return value (despite an output schema), default behaviors, or edge cases. The agent lacks sufficient context to invoke the tool reliably.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the tool description adds no meaning to the four parameters (path, start_line, end_line, repository_id). It only hints at line range semantics but does not explain repository_id or path, leaving the agent reliant on parameter names alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Read') and resource ('repository file'), and specifies it reads a bounded line range, which conveys a specific scope. However, it does not explicitly distinguish from sibling tools like repository_map or search_symbol, missing the chance to clarify when to use this tool over alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus its siblings (repository_map, search_symbol, find_references). The description does not mention prerequisites, context, or exclusions, leaving the agent to infer usage without clear direction.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_referencesC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It does not disclose whether the tool is read-only, safe, or requires specific permissions. The term 'indexed' implies a safe operation, but this is not explicit.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, making it concise. However, it omits important behavioral and usage details, so it is not appropriately sized for the tool's complexity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  An output schema exists, so return value details are not required. However, the description lacks guidance on when to use the tool and what the output represents (e.g., file locations or code snippets). Adequate but not comprehensive.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%. The description adds no information about the two required parameters (repository_id, symbol_name) beyond their names, which are self-explanatory but still lacking context.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Find indexed call sites for a symbol' clearly states the verb (find) and the specific resource (indexed call sites). It differentiates from sibling tools like repository_map, search_symbol, and read_file by focusing on call site retrieval.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives. For example, it does not explain when to prefer find_references over search_symbol or how it differs from repository_map.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• repository_mapB
  3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It suggests a read operation by using 'return', but does not explicitly state read-only behavior, permissions, or potential side effects. Minimal disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that is concise and directly states the tool's purpose. Every word is necessary, and it is front-loaded with the key action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has a single parameter and an output schema, the description is somewhat complete but lacks details like what 'files and symbols' specifically means and whether the repository must be indexed. The output schema may cover return format, but the description could be more informative for an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description must add meaning but it does not. The only parameter 'repository_id' is not explained beyond its type. No additional context about the parameter's role or constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns files and symbols from an indexed repository, which distinguishes it from siblings like search_symbol (search for symbols) and read_file (read a file). The verb 'return' and resource 'files and symbols from an indexed repository' are specific.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like search_symbol or find_references. The description lacks any context about use cases or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.