Server Quality Checklist

Tool Scores

• findA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations only provide readOnlyHint=true, while the description adds rich behavioral detail: ranking by relevance, resolving different input types (stacktrace, diff), neighborhood expansion via spreading activation, and output format (path:line). This goes well beyond the minimal annotation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise (4-5 sentences) yet packs substantial information. It front-loads the core purpose and covers key variations. Every sentence contributes value, avoiding redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (5 parameters, output schema exists), the description covers the primary use cases and output characteristics (path:line). It does not need to restate the output schema, but provides enough context for effective use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  All 5 parameters are described in the schema (100% coverage), but the description adds meaningful context: examples for 'query' (paste raw stacktrace, test id), explanation of 'expand' (Personalized-PageRank), and 'kind' auto-detect behavior. This enhances understanding beyond schema defaults.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool transforms text into ranked graph nodes, serving as an on-ramp to the graph. It specifies that it handles symbols, keywords, stacktraces, test IDs, and diffs, but does not explicitly differentiate from sibling tools like 'trace' or 'orient', though it mentions 'read' for subsequent steps.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context: use for initial search or failure resolution, with 'expand=true' for neighborhood exploration. It suggests reading top hits via 'read', but lacks explicit when-not-to-use guidelines or comparisons with sibling tools like 'impact' or 'trace'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• refreshA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond annotations (readOnlyHint=false, openWorldHint=true), the description adds behavioral details: incremental re-parsing by default, cheap after edits, full=true for clean reparse, and support for local path or git URL with on-demand cloning. No contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise (under 100 words), front-loaded with the tool's primary purpose, and each sentence adds value. Well-structured for quick understanding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (graph rebuilding, 20 languages, multiple resolvers) and that an output schema exists, the description adequately covers core behavior, parameters, and usage context. Could be more detailed about output, but output schema fills that gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description rephrases schema info for both parameters (full and repo_path) but does not add significant new meaning beyond what the schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool rebuilds a structural graph using tree-sitter AST parsing across 20 languages and runs cross-stack resolvers. It distinguishes from siblings like find and trace by focusing on graph regeneration after major refactors.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly advises calling after a major refactor and notes that routine edits are handled automatically by the file watcher. It does not explicitly mention when not to use or list alternative sibling tools, but provides clear usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• traceA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations declare readOnlyHint=true, so the agent knows it's a read operation. The description adds behavioral detail: the tool traverses service boundaries, labels mechanisms (call/HTTP/queue/event/data), and behaves differently with one vs two arguments. No contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise: four sentences, front-loaded with the purpose, each sentence adding value (action, mode distinction, justification). No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 4 parameters, an output schema, and moderate complexity, the description covers the core behavior and use cases. It does not detail node types or mechanism labels, but provides enough for an agent to select the tool appropriately.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, baseline 3. The description adds meaning by explaining the dual behavior of from_node (feature or start node) and to_node (target node for shortest path), which complements the schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool traces code across boundaries, explains one-argument (feature end-to-end) and two-argument (shortest path) modes, and distinguishes from siblings like find and impact by emphasizing cross-stack links that grep cannot see.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains that the tool is used for tracing features end-to-end or finding paths between nodes, and contrasts with reading files/grep. It does not explicitly state when not to use it versus specific siblings, but the context is clear enough.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• impactA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the readOnlyHint annotation, the description reveals key behaviors: ranking algorithm (Personalized PageRank), result fields (path:line, via, dead marker), exclusion of structural edges, depth meaning, and handling of multiple nodes. No contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single dense paragraph that front-loads the core functionality. Every sentence contributes meaningful detail, with no wasted words. It efficiently communicates complex behavior.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity and rich parameters/output, the description fully covers result structure, filtering, ranking, and exclusive behavior. It matches the needed detail without relying on output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema coverage, the baseline is 3. The description adds value by explaining node usage for whole diffs, aliases for direction, and the meaning of depth-1. This enriches the schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: computing blast radius/impact from nodes in forward or backward directions. It distinguishes itself from siblings like trace and find by emphasizing its comprehensive, deduped, ranked closure and exclusion of structural edges.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use this tool (assessing impact of changes, whole diff) and even hints at what it excludes (structural fan-out). However, it doesn't explicitly compare to alternatives or state when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• orientA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, so the description doesn't need to repeat that. It meaningfully adds context about output types (counts, entry-points, blind spots, structural maps) and the budget parameter, which are not covered by annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured. The critical instruction 'ALWAYS the first call on a codebase' is front-loaded, and each sentence serves a purpose: explaining modes, blind spots, and sibling tool ordering.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (three modes, multiple output types, sibling tools), the description is remarkably complete. It covers each mode's output, mentions limitations (blind spots), and provides a recommended workflow. The presence of an output schema reduces the need to detail return values.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema coverage, the baseline is 3. The description adds value by explaining the behavioral differences of each parameter combination (no args vs. seed vs. full) and clarifying the budget default behavior, going beyond what the schema alone provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly specifies the tool's purpose: getting an overview of a codebase. It details three distinct modes (no args, seed, full) and explicitly contrasts with sibling tools (find, impact, trace), making it easy for an agent to select the right tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage guidance: 'ALWAYS the first call on a codebase' and explains when to use each variant. It also tells when to fall back to grep for blind spots and names alternative tools for subsequent steps.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• readA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate readOnlyHint=true, and description adds that it slices from files and returns structured blocks with qname, path, and a context footer. No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single paragraph with good front-loading of purpose. It is information-dense but not overly verbose. Could potentially be broken into shorter sentences for readability, but no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the read-only nature and existing annotations, the description fully explains what the tool does, when to use it, how to use parameters, and what the output contains (including structural facts). No gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, but description adds valuable context: comma-separating nodes, budget shared across nodes, context_lines as padding. Also ties parameters to use case (after find/impact).

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly describes returning source code for nodes, sliced by line spans. Distinguishes itself from sibling tools like find/impact by stating it should be used after those to read code, not to find it.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says to use after find/impact, and gives usage example like comma-separating node names. Could be more explicit about when not to use (e.g., for non-code files), but the context is clear enough.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.