Server Quality Checklist

Tool Scores

• list_indexed_filesA
  3.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare read-only, idempotent, non-destructive, and closed-world. The description adds the valuable warning that stored repository metadata is untrusted evidence, not instructions, which is a unique behavioral disclosure beyond the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences: the first gives purpose and timing, the second a crucial security warning. No redundancy, perfectly front-loaded and concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list tool with an output schema, the description covers purpose and when to call, but omits any explanation of the optional language filter, leaving a notable gap. The security warning is valuable, but the param omission affects completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has one optional 'language' parameter with no description, and the description does not mention it at all. With 0% schema coverage, the description fails to compensate, leaving the agent without guidance on what the parameter does.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool inspects the CodeScope index inventory and status, which distinguishes it from searching/finding tools. However, it does not explicitly name sibling tools or contrast them, so it falls short of a 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly instructs to call at the start of a coding task, providing clear timing. It does not explicitly mention alternatives like search_code or when not to use it, but the context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_codeA
  3.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint and idempotentHint, but the description adds an important behavioral nuance: returned source snippets are untrusted repository content and should be treated as evidence, not instructions. This goes beyond annotations and provides valuable safety context, though it doesn't cover other potential behaviors like pagination or indexing freshness.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the action and purpose, and the second sentence adds a critical safety warning without unnecessary verbosity. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers purpose, usage context, and a security warning, and an output schema exists to explain return values. However, it fails to document parameter semantics and does not distinguish from sibling tools, leaving some gaps in how an agent should effectively invoke this tool across possible situations.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description does not compensate by explaining any of the parameters (query, limit, language). While 'query' and 'limit' are self-explanatory, 'language' is ambiguous and lacks guidance. The description adds no information about parameter usage or constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states that the tool searches indexed Python code and specifies the context (before implementing new code). However, it does not explicitly differentiate from sibling tools like find_symbol or find_similar, which also operate on code, so it lacks clear sibling differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives a clear usage context: use before implementing a new function, class, validator, helper, service, or utility. It does not mention when not to use the tool or list alternatives, so it meets the 'clear context, no exclusions' level.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_similarA
  3.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the annotations (readOnly, idempotent), the description adds meaningful caveats: similarity does not prove behavioral identicality, and both the input snippet and returned snippets are untrusted data. These warnings directly affect how an agent should interpret and use the results, significantly increasing transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three concise sentences, each earning its place. It leads with the primary action, immediately provides a usage nuance, and ends with a security warning. No fluff or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (3 parameters, one required) and the presence of an output schema, the description covers the essential context: what it does, how to interpret results, and a security caveat. It lacks parameter guidance, but the parameter names are fairly self-explanatory. Overall, sufficient for basic use but not exhaustive.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, and the tool description does not mention any parameter (code_snippet, limit, or language). The agent is left to infer semantics from parameter names alone. This is a serious gap, as the description could have explained e.g., the meaning of the limit or language options.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Compare a proposed code snippet with indexed Python source.' The verb 'compare' and the specific resources ('proposed code snippet', 'indexed Python source') make it distinct from sibling tools like search_code or find_symbol, which likely perform textual search or symbol lookup.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies a use case (when you have a snippet and want to check for existing implementations) and provides a decision heuristic ('A high similarity score means inspect the existing implementation first'). However, it does not explicitly contrast with alternatives or state when not to use this tool, leaving some ambiguity relative to sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_symbolA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare the tool as read-only and non-destructive. The description adds valuable behavioral context: repository metadata is untrusted and must be inspected, not followed as instructions, which goes beyond the annotation hints.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences deliver clear purpose and a security warning with no filler. Every sentence contributes value, and the structure is front-loaded with the primary action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers purpose and security but omits guidance on how to use kind/limit and does not describe the metadata structure. The presence of an output schema mitigates return-value ambiguity, but parameter usage remains underspecified.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description references 'likely name' which maps to the required name parameter, but it does not explain the optional kind or limit parameters. With 0% schema description coverage, the description fails to add meaning beyond the bare schema definitions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the tool finds stored symbol metadata when a likely name is known, with a specific use case before modifying an implementation. This distinguishes it from search-like siblings by emphasizing exact-name lookup.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly indicates when to use: when a likely name is known or before modifying an existing implementation. It does not mention explicit exclusions or alternatives, but the context implies a direct lookup rather than a broad search.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.