Server Quality Checklist

Tool Scores

• pathC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of disclosing behavior. It only states the core action but does not explain what happens when no path exists, whether the path is returned as a list of symbols or edges, or any error conditions. This leaves significant behavioral ambiguity.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence with no redundant words. It immediately communicates the action and resource, making it easy to parse and understand.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity, the description still lacks essential details for correct invocation and interpretation: the return value format, error handling for disconnected nodes, and input format requirements. With no output schema or annotations, the description alone is insufficient for complete understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has no descriptions for 'from' and 'to', and schema description coverage is 0%. The description adds only that these are 'symbols', but does not clarify expected formats (e.g., fully qualified names, file paths), constraints, or definitions. This provides minimal value beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('Trace') and a distinct resource ('shortest path between two symbols in the codebase graph'). This differentiates it from sibling tools like 'search' and 'node', which focus on other aspects of the graph.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. The description does not mention any prerequisites, exclusions, or scenarios where another tool would be more appropriate, leaving the agent to infer usage solely from the tool's name.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• nodeA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It reveals a key behavioral trait: the tool returns line ranges specifically to enable targeted reading, avoiding reading whole files. However, it does not explicitly state whether the operation is read-only, nor does it describe error behavior or depth parameter effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with purpose and then usage. No unnecessary words, every clause adds information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description provides a clear workflow and the primary return type (file:line ranges), but lacks an explicit return structure and does not explain the 'depth' parameter. Given the lack of annotations and output schema, the description is not fully complete for an agent to invoke correctly without guessing depth semantics.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has zero description coverage, so the description must explain parameters. It implicitly covers 'name' via 'a symbol', but completely omits 'depth' (an integer with default 2). The description adds zero meaning to the depth parameter, which could represent traversal depth or recursion level, leaving the agent to guess.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: retrieving a symbol's connections (callers, callees, imports) with file:line ranges. It uses a specific verb 'Get' and a specific resource 'symbol's connections', and it distinguishes itself from sibling search and impact by focusing on dependency relationships rather than discovery or impact analysis.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly instructs to use AFTER search and then use built-in Read with the returned line ranges. It also positions itself as an alternative to reading whole files, which provides clear usage context. However, it doesn't explicitly state when not to use it, but the conditional 'after search' gives guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• local_filesA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It explicitly flags the operation as 'EXPENSIVE' and clarifies it returns full files, not line ranges. While it doesn't cover error handling or permissions, the key behavioral traits (read-only, cost, full-file scope) are disclosed meaningfully.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, each serving a distinct purpose: stating the core action, warning about cost, and providing usage guidance. It is concise, front-loaded, and free of filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple two-parameter tool with no output schema, the description covers the core purpose, cost, and alternatives. It could detail path semantics or error scenarios, but given the simplicity, it is sufficiently complete for an agent to select and invoke correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% and the description does not explain the 'paths' or 'max_bytes' parameters. Although the parameter names are somewhat self-explanatory and the schema includes a default for max_bytes, the description adds no additional meaning or guidance beyond what the schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool reads full source files from disk, using the specific verb 'Read' and the resource 'full source files'. It also distinguishes itself from alternatives by noting it's for whole files only, contrasting with line-range reads from search/node results.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit guidance is provided: 'Prefer built-in Read with line ranges from search/node results instead' and 'Use this only when you need a whole file that search/node didn't cover.' This clearly specifies when to use and when not to use the tool, naming an alternative.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• impactA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It states that traversal is exhaustive, that ALL edge types are considered, and provides a negative guarantee about files not listed. This adds meaningful context about the tool's behavior and scope, though it could mention potential performance or output size implications.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded: the first sentence states the core purpose, followed by scope, output, usage, and a key guarantee. Each of the five sentences adds essential information with no redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter tool with no output schema and no annotations, the description covers the essential points: what it does, when to use it, what it returns, and a completeness guarantee. It could be slightly more explicit about the exact input format or result details, but it is largely complete for its complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema only defines 'name' as a string with no description, and schema coverage is 0%. The description compensates by implying that 'name' refers to the symbol being changed (e.g., 'changing a symbol' and 'the target'). While not explicitly stating 'the name parameter is the symbol', the context makes it clear, adding meaning beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: computing the blast radius of changing a symbol, with specific behavior (exhaustive traversal of ALL edge types) and output (every affected file with symbols to check). This clearly distinguishes it from sibling tools like search, node, path, and local_files, which are not about impact analysis.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use the tool ('Use before refactoring') and includes a useful note about what files are not listed ('Files not listed do not depend on the target'). However, it does not mention when not to use it or name any alternative tools, so it falls short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• searchA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses return format ('name, kind, file path with line ranges (file:start-end), and confidence [H/M/L]'), demonstrates handling of exact names, paths, and natural language, and explains the recommended follow-up action for retrieving source. This goes well beyond the schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is four sentences, each serving a distinct purpose: purpose, query flexibility, output format, and usage guidance. No wasted words; front-loaded with the core purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite lacking an output schema and annotations, the description explains what results look like and how to retrieve source code. It covers the search query types and the near parameter is adequately described in the schema. The description is sufficient for an agent to select and invoke the tool effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds rich meaning to the query parameter by illustrating exact names, file paths, and natural language queries. Since schema description coverage is only 50% (query lacks schema description), the description fully compensates by explaining query semantics in detail.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Search the codebase for symbols, files, or concepts.' It provides concrete examples of acceptable query formats and explicitly positions itself as a replacement for grep/glob, distinguishing it from sibling tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit usage priority: 'Use this FIRST — replaces grep and glob.' Also advises how to follow up: 'Use built-in Read with offset=line_start, limit=line_end-line_start to get source.' This gives clear when-to-use guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.