Server Quality Checklist

Tool Scores

• read_interfaceA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It describes what the tool returns (interface elements) and its non-destructive nature (implied by 'read'), but doesn't cover aspects like error handling, performance characteristics, or authentication needs. The description adds useful context about the tool's scope but lacks comprehensive behavioral details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (purpose, returns, usage context, inputs, example) and uses emojis for visual organization. While slightly longer than minimal, every sentence adds value. The information is front-loaded with the core purpose stated first.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (2 parameters, no annotations, but has output schema), the description provides good coverage. It explains the purpose, parameters, and includes an example. Since an output schema exists, it doesn't need to detail return values. The main gap is lack of behavioral details like error cases or limitations.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage for 2 parameters, the description compensates well by explaining both parameters in the INPUTS section: 'root_path' as 'The ABSOLUTE path to the project root' and 'file_path' as 'The path to the specific file you want to read (can be relative to root)'. It also provides a concrete example that clarifies usage. This adds significant meaning beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('Get a high-level overview', 'understanding how to USE') and distinguishes it from siblings by focusing on extracting interface information rather than exploring, finding symbols, or analyzing breaks. It explicitly mentions what it returns (function signatures, class definitions, docstrings) and what it doesn't do ('without reading implementation').

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context about when to use this tool ('Perfect for understanding how to USE a module without reading the whole thing'), which implicitly suggests alternatives like reading the full file. However, it doesn't explicitly mention when not to use it or name specific sibling tools as alternatives, though the context is sufficient for differentiation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_symbolA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden and adds valuable behavioral context: it explains the fuzzy matching capability, returns a list of symbol objects or empty list if no matches, and specifies that results should be saved for use with what_breaks(). It doesn't cover permissions or rate limits, but provides clear operational details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured with clear sections (description, usage, inputs, examples, returns, next steps) and front-loaded purpose. Slightly verbose due to detailed examples and instructions, but every sentence adds value for tool invocation and workflow integration.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 2 parameters with 0% schema coverage, no annotations, but an output schema exists, the description is complete: it covers purpose, usage, parameters with semantics, example inputs/outputs, return behavior, and integration with sibling tools (explore_repo and what_breaks), leaving no gaps for agent operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate fully. It does so by explaining both parameters: root_path ('Same ABSOLUTE path used in explore_repo') and query ('What you're looking for (fuzzy search works!)') with examples and formatting guidance, adding meaning beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Find specific functions, classes, or methods in the codebase' with 'fuzzy matching'. It distinguishes from siblings like explore_repo (which it follows) and what_breaks (which it precedes), making the verb+resource+scope specific and differentiated.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit guidance is provided: 'USE THIS AFTER explore_repo() when you need to locate a specific piece of code.' It names the sibling tool explore_repo as a prerequisite and indicates when to use this tool (for fuzzy searching after exploration), with no misleading or missing exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• what_breaksA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes key behaviors: it performs structural search using ast-grep, ignores comments and strings, returns 2 lines of context around each match, and outputs a structured result with references and metadata. However, it doesn't mention potential limitations like performance or error handling.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (purpose, usage, improvements, input, examples), but it includes an extensive example input and output that could be condensed. Most sentences earn their place by providing essential information, though the examples are detailed and might be overly verbose for a concise description.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (structural code analysis), no annotations, and an output schema that covers return values, the description is complete enough. It explains the tool's purpose, usage context, behavioral traits, parameter requirements, and provides illustrative examples, compensating well for the lack of structured metadata.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It adds significant meaning beyond the schema by explaining that 'exact_symbol' must be 'THE ENTIRE SYMBOL OBJECT from find_symbol(), not just the name!' and specifying it 'Must be a dictionary with AT LEAST 'name' and 'path' keys.' This clarifies the parameter's purpose and constraints that aren't in the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'See what code might break if you change this symbol.' It specifies the verb ('see what code might break') and resource ('this symbol'), and distinguishes it from sibling tools by explicitly mentioning it should be used 'AFTER find_symbol()' and contrasting with 'structural search' versus other approaches.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool: 'USE THIS AFTER find_symbol() to understand the impact of changing a function/class.' It also specifies an alternative approach by noting it 'Uses structural search (ast-grep) to find ACTUAL code references (ignoring comments/strings),' implying a distinction from other search methods.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• explore_repoA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure and excels. It explains the tool's progressive workflow, caching behavior ('Symbol extraction is cached per git commit - subsequent calls are instant!'), performance implications, and output format through detailed examples. It also clarifies path requirements ('ABSOLUTE path... NOT relative paths') and default behaviors.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (workflow, inputs, examples, tips, next steps) and uses emojis for visual organization. While slightly verbose, every sentence adds value: the workflow guides usage, examples illustrate outputs, tips optimize performance, and next steps connect to sibling tools. It could be more concise but remains highly effective.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is exceptionally complete for a tool with 5 parameters, 0% schema coverage, no annotations, but an output schema. It covers purpose, usage workflow, parameter details, behavioral traits (caching, performance), examples with output formats, and integration with sibling tools. The presence of an output schema means return values don't need explanation, and the description fills all other gaps comprehensively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Given 0% schema description coverage, the description fully compensates by providing comprehensive parameter semantics. Each of the 5 parameters is clearly explained with purpose, constraints, and examples: root_path (absolute vs. relative), max_depth (unlimited vs. limited), include_symbols (dirs only vs. with symbols), focus_dirs (top-level directories to filter), and max_symbols_per_file (limit when symbols shown). The examples demonstrate practical usage of all parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Map the codebase structure - start simple, then zoom in!' It specifies the verb ('explore', 'map') and resource ('repo', 'codebase structure'), and distinguishes it from sibling tools by focusing on structural discovery rather than symbol searching (find_symbol), interface reading (read_interface), or breakage analysis (what_breaks).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when and how to use this tool versus alternatives. It outlines a 'PROGRESSIVE DISCOVERY WORKFLOW' with three steps, advises starting with 'include_symbols=False' to avoid overload, and directs users to 'use find_symbol() to locate specific functions' after exploration. It clearly differentiates from sibling tools by positioning explore_repo as the entry point for structural mapping.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.