Server Quality Checklist

Tool Scores

• search_codeC
  2.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description bears full responsibility for behavioral disclosure. It only states the output type (ranked snippets with file locations) but omits any mention of side effects, authentication requirements, rate limits, or whether the operation is read-only.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single 10-word sentence, which is concise but lacks substantive detail. It earns its space but could be expanded with key information without becoming verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the presence of 5 parameters (1 required), 0% schema coverage, and no annotations, the description is insufficiently complete. It does not cover query syntax, result ordering, or filtering options, leaving significant gaps for an agent to correctly invoke the tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% description coverage for its parameters, meaning none are documented in the schema. The tool description fails to explain any parameter (query, k, repo, language, path_prefix), leaving the agent with no semantic guidance beyond parameter names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'search', the resource 'indexed code', and the outcome 'return ranked snippets with file locations'. It effectively distinguishes from sibling tools like 'get_symbol' (which retrieves a single symbol) and 'list_repos' (which lists repositories).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It does not mention typical use cases, prerequisites, or scenarios where other tools would be preferred.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• reindexC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, and the description only says 'Reindex a repo' without disclosing behavioral traits such as idempotency, required permissions, or potential side effects. The description does not add enough context beyond the name.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise (one short sentence), which is front-loaded but insufficiently sized to convey necessary details. While concise, it sacrifices completeness for brevity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the low schema coverage and no annotations, the description should provide more context about the tool's behavior and output. Although an output schema exists (not shown), the description does not reference it or explain what happens after reindexing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description is responsible for explaining parameter meaning. It only mentions that path is optional under a condition, but does not describe what the repo_path value represents or its format, limiting clarity.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action (reindex) and the resource (a repo), and adds a condition about when the path parameter is optional. This distinguishes it from sibling tools which handle symbol lookup, repo listing, and code search.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives a rule for when the path is optional, implying usage context when there is exactly one indexed repo. However, it does not provide guidance on when to use this tool versus alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_symbolC
  2.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description carries full burden and discloses the fallback behavior. However, it omits other behaviors such as what happens on failure, authentication needs, or rate limits. Some transparency is provided, but not comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is one sentence with no unnecessary words. It front-loads the core action and is appropriately sized for the tool's complexity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has two parameters and no nested objects, the description should at least clarify parameter usage. It fails to do so, and while an output schema exists, the lack of parameter semantics makes it incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, yet the description does not explain the parameters 'name' (required) or 'repo' (optional). The user is left to guess their purpose, which is insufficient for effective tool use.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool looks up a symbol from indexed metadata with a fallback to search. It distinguishes from sibling 'search_code' by implying a direct lookup then fallback. Could be more precise about what 'indexed metadata' means.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no explicit guidance on when to use this tool versus alternatives like 'search_code' or 'list_repos'. The fallback mention is implicit, but no when-not-to-use or prerequisites are stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_reposA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden for behavioral disclosure. It only states the tool lists repos and freshness state, with no mention of side effects, authentication needs, rate limits, or what 'freshness state' entails. This is insufficient for a transparent tool definition.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise—one sentence with clear verb and resource. It is front-loaded and contains no unnecessary words, earning every word's place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (no parameters, output schema exists), the description is mostly complete. However, it could benefit from clarifying 'freshness state' (e.g., last indexed time) or output structure, but the output schema likely covers that. For a list tool, this suffices.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are no parameters, so the input schema is trivially covered. Per guidelines, 0 params yields a baseline of 4. The description adds no parameter info, but none is needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists indexed repos and their freshness state. The verb 'list' and resource 'indexed repos' are specific. It distinguishes from siblings like 'get_symbol' (lookup a symbol), 'reindex' (trigger indexing), and 'search_code' (search within code), as listing repos is a distinct operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. Sibling tools exist (get_symbol, reindex, search_code) but no comparative context is given, leaving the agent to infer usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.