Server Quality Checklist

Tool Scores

• search_symbolsB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden but only provides a high-level action. It does not disclose pattern syntax (regex, glob, substring), result format, pagination, or behavior when directory isn't indexed. This is a significant transparency gap for a search tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no filler. It efficiently communicates the core action and scope.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is too sparse for a tool with 4 parameters and no output schema. It omits critical context like pattern syntax, return payload expectations, prerequisite indexing, and how it differs from search_code. These gaps would force an agent to guess.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already describes all 4 parameters with 100% coverage, so the baseline is 3. The description's phrase 'by name or pattern' marginally reinforces the query parameter but adds no new semantic insight beyond what the schema provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function with a specific verb ('Find') and resource ('symbols in an indexed codebase'), listing example symbol types. It distinguishes itself from siblings like search_code (which likely searches code content) and index_codebase (which creates the index).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the codebase must be indexed but gives no explicit when-to-use or alternative guidance. It does not reference sibling tools or mention when to prefer symbol search over code search, so usage context is only implied.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• index_codebaseA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It does not disclose whether indexing has side effects (e.g., writes files), whether it modifies the directory, or its resource/performance implications. This is a significant transparency gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single sentence that is front-loaded with the core action and outcome, followed by supported languages. Every word earns its place; no redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter tool with no output schema, the description conveys the essential function. However, it omits behavioral context (side effects, whether it must be run before sibling search tools) and does not reference the tool's place in the workflow, making it only minimally complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, and the parameter 'directory' is adequately described as 'Absolute path to the project root directory to index'. The description adds no additional meaning beyond the schema, which is acceptable given the high coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('parse') and resource ('directory into symbols and chunks') and clearly states the outcome ('building a searchable index'). It also lists supported languages, which distinguishes it from sibling search/outline tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage as a precursor to search tools by stating it builds a searchable index, but it does not explicitly say when to use it versus siblings or mention that search tools depend on it. No alternatives or exclusions are given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_file_outlineA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the burden of behavioral disclosure. It indicates a read-only operation ('Get') and specifies the output (structure with line numbers), but it does not disclose potential requirements such as the file needing to be indexed, error behavior, or whether the tool reads from a cached index. Basic transparency, but lacking depth.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that states the action and the expected output. Every word earns its place, with no redundancy or fluff, making it concise and easily scannable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with two required parameters and a clear output specification, the description is largely complete. It explains what the output contains, which is necessary since there is no output schema. However, it could mention that the file must be part of an indexed project, though this is covered by the parameter descriptions. Overall, sufficient for the tool's simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already fully describes both parameters (filePath and directory) with meaningful descriptions, resulting in 100% schema coverage. The tool description adds no additional parameter semantics beyond what the schema provides, so the baseline score of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Get the structure of a file' and lists the specific elements (functions, classes, imports, exports) with line numbers. This distinguishes it from sibling tools like search_symbols or search_code, which focus on searching rather than providing a structural outline.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is used when a structural overview of a file is needed, but it does not explicitly state when to use this tool versus alternatives, nor does it mention any exclusions or prerequisites. There is no guidance on when not to use it or how it differs from search_code.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_project_summaryA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of explaining behavior. It discloses what the overview includes and implies a read-only operation. The 'indexed project' requirement provides useful context, though it does not detail error handling for non-indexed or invalid paths.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single, front-loaded sentence with no wasted words. It conveys the action, resource, and specific content in a clear and efficient manner.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simplicity of the tool (one parameter, no output schema), the description adequately covers the purpose and expected return contents. It could mention outcomes for non-indexed directories, but overall it is sufficient for agent use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema fully describes the only parameter (directory) as 'Absolute path to the indexed project' with 100% coverage. The tool description adds no additional param-level insights, so the baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Get an overview of an indexed project' and enumerates specific content: tech stack, file counts by language, and directory structure. This distinguishes it from sibling tools that search code or outline individual files, making the purpose unmistakable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when a project-level overview is needed and mentions the prerequisite that the project must be indexed. It does not explicitly contrast with siblings or list exclusions, but the context is clear enough for an agent to know when to invoke it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_codeA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses the BM25 ranking algorithm, indicates the search is across all files, and specifies the output ('ranked results with matching lines'). This adds meaningful behavioral context beyond the tool name, though it omits error cases or performance implications.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no filler. The first sentence front-loads the core purpose, and the second covers the output format. Every word contributes useful information, making it appropriately concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has no output schema, so the description's mention of 'ranked results with matching lines' is valuable. The schema covers parameters, and the description covers core behavior and the indexing prerequisite. However, it does not address what happens if the directory is not indexed or how maxResults behaves, leaving minor gaps. Still sufficient for a search tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% for all three parameters (query, directory, maxResults). The description does not add parameter-specific details beyond what the schema already explains; the mention of 'all files' and 'indexed codebase' broadly maps to directory, but this is not a substantial addition. Therefore, baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states 'BM25 full-text search across all files in an indexed codebase,' which is a specific verb (search) + resource (all files) + scope (indexed codebase). This clearly distinguishes it from sibling tools like search_symbols (which searches symbols) and index_codebase (which indexes).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'in an indexed codebase' implies a prerequisite that index_codebase should be run first, and 'full-text search' signals this is for content-based lookup rather than symbol search. However, it does not explicitly name alternatives or state when not to use the tool, so it stops short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.