Server Quality Checklist

Tool Scores

• exploreC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It only lists query categories and does not mention whether the operation is read-only, what the output format is, how the question, anchor, and relation parameters interact, or any potential side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise phrase with no filler or redundant information, and it front-loads the key concept. However, its extreme brevity borders on under-specification, though conciseness itself is strong.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 3 parameters, 0% schema coverage, no annotations, and no output schema, this description is far too minimal. It omits the required question semantics, anchor behavior, relation default, and expected response, making it inadequate for reliable tool invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate by explaining the parameters. It only implicitly references the relation enum values (callers, callees, dependencies, impact, architecture overview) and says nothing about the required question parameter or the optional anchor parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the tool handles 'Structural questions' and lists specific topics such as callers, callees, dependencies, impact, and architecture overview, which makes its domain clear and distinguishes it from siblings like locate and search. However, it is a noun phrase rather than an explicit verb-plus-resource construction, so the action statement is slightly weaker.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies that the tool should be used for structural or architectural questions by enumerating relation types, but it does not explicitly state when to use this tool versus the sibling tools locate, read, or search. It also lacks any exclusions or alternative guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• readC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The phrases 'block-expanded and size-bounded' provide some behavioral hints, but they are unexplained and cryptic. With no annotations, the description does not adequately disclose side effects, permissions, or return format for a read operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, front-loaded with the verb, and contains no unnecessary words. It is highly concise, though the extreme brevity sacrifices some clarity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 3-parameter tool with no output schema and no annotations, the description is incomplete. It lacks details on return format, usage scenarios, and does not define the purpose of all parameters, relying on cryptic hints.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description only partially clarifies the 'target' parameter via 'symbol or file:start-end range'. The 'expand' and 'max_tokens' parameters are not explicitly explained; the cryptic 'block-expanded' and 'size-bounded' are insufficient compensation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Read a symbol or file:start-end range' clearly states the verb (read) and resource (symbol or file range). It distinguishes from sibling tools like locate/explore/search, which are discovery-oriented rather than retrieval-oriented.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no guidance on when to use this tool versus alternatives, no mention of prerequisites, exclusions, or suggested context. The description is purely definitional and offers no usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• searchB
  3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations available, the description carries the full burden of disclosing behavior. It only says 'always filtered and gated,' which is vague and doesn't explain safety (e.g., read-only nature), what filtering entails, or what gating means. This is insufficient for a search tool that might access sensitive files.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that packs the core purpose, a usage guideline, and behavioral hints without wasted words. It is front-loaded with the primary action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  This tool has 9 parameters, no annotations, and no output schema, but the description does not explain return values, parameter semantics, or access constraints. It gives only a high-level hint about filtering and gating, leaving significant gaps for a complex search tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% and the description adds minimal parameter meaning. It mentions 'keyword or regex' which hints at the 'pattern' and 'regex' parameters, but the other 7 parameters (scope, exclude, max_hits, force, etc.) are completely unexplained.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states this is a keyword or regex search tool, and labels it a sanctioned replacement for grep. This distinguishes it from sibling tools like locate (file location) and read (file contents), though it doesn't explicitly state what is searched (file content is implied by grep).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'sanctioned replacement for grep' provides an explicit usage context and implicitly tells the agent to use this instead of grep. However, it doesn't explicitly contrast with sibling tools (locate, read, explore) or state when not to use this tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• locateA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the transparency burden. It discloses that the tool returns locations only and excludes source bodies, which is a key behavioral trait. It does not mention other traits like side effects or rate limits, but for a locate operation this is reasonably transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence that front-loads the action and clarifies output with a dash. Every word adds value, with no redundant content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 4 parameters, no annotations, and no output schema. The description covers the essential purpose and return type, but leaves parameter semantics unexplained and does not describe the location format. It is adequate for a simple tool but has clear gaps for full completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description does not explain the parameters. It only alludes to 'symbol or concept' which relates to the query, but kind, scope, and max_results are left unexplained. The description fails to compensate for the lack of schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's action (find) and resource (where a symbol or concept is defined or referenced), and explicitly notes it returns only locations, not source bodies. This distinguishes it from siblings like read and search, which would provide content or broader search results.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use the tool: when you need to locate definitions or references without needing source bodies. However, it does not explicitly name alternatives or state when not to use it, so it stops short of full exclusion guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.