Server Quality Checklist

Tool Scores

• index_repoA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It states the action but does not disclose side effects, idempotency, persistence, required permissions, or whether the operation is synchronous. The mutation of the repository index is implied but not elaborated.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise and front-loaded. The first sentence states the purpose, and the second documents the parameter. No unnecessary words or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple (one parameter) and has an output schema, so return value documentation is not necessary. However, the description lacks guidance on when to use the tool relative to siblings, does not explain the indexing behavior (e.g., overwriting vs incremental), and omits any potential side effects or requirements. It is minimally viable but leaves gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides only the parameter name and type, but the description adds meaningful context: 'absolute path to the repo root to index.' This clarifies the expected format and purpose of the path parameter, compensating for the lack of schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function with a specific verb ('ingest') and resource ('a repository'), and specifies the output ('build the hybrid (dense + BM25) index'). This distinguishes it from sibling tools like search_code, which are for search operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies that indexing should happen before searching (given the sibling tool search_code), but it does not explicitly state when to use this tool versus alternatives. There is no direct guidance on prerequisites or ordering.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• impact_radiusA
  4.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It states the primary action (list callers) but does not disclose important traits such as whether callers are direct or transitive, whether results are limited to the indexed repo, or potential performance implications. This ambiguity leaves the agent guessing about blast radius scope.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is exceptionally concise: two sentences plus a one-line args annotation. It fronts the purpose and adds necessary parameter context without any fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter tool with an output schema, the description covers invoking it correctly. It could be improved by clarifying the semantics of 'callers' (direct vs transitive), but overall it is sufficiently complete for basic use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description thoroughly explains the only parameter chunk_id, including an example format ('pkg/mod.py::ClassA.method') and its source ('from search results'). This adds significant meaning beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: given a chunk id, list callers (the blast radius). This is a specific verb+resource that distinguishes it from sibling tools like search_code or get_context.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description indicates usage context: 'from search results' and references 'the qualname shown in results', guiding when the tool is appropriate. It does not explicitly name alternatives or exclusion cases, but context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_codeA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It provides useful context on parameter behavior (hybrid mode, rerank trade-offs) and even discloses that rerank is slower and requires the '[embeddings] extra'. It does not, however, mention any side effects, permissions, or limitations beyond those parameter-specific notes.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficiently structured as a docstring with a one-sentence purpose followed by a concise Args section. Every sentence earns its place, explaining each parameter without redundancy or fluff, making it easy for an agent to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the presence of an output schema, the description appropriately focuses on input parameters and behavioral specifics like speed/dependency. It covers the core functionality and parameter semantics well, but it lacks explicit prerequisites (e.g., 'repo must be indexed first') or typical error scenarios, leaving a small gap in contextual completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Despite 0% schema description coverage, the description compensates fully by explaining every parameter: query (natural-language or keyword), k (number of results), hybrid (dense vs dense+BM25 fusion), and rerank (wide pool reordering with cross-encoder, with precision/speed trade-off and dependency). This adds significant meaning beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Search') with a defined resource ('the indexed codebase') and a clear outcome ('return the top matching definitions'). This clearly distinguishes it from sibling tools like index_repo (indexing), status (status checks), get_context (retrieval), and impact_radius (analysis).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when you need to search code, but it does not explicitly state when to use this tool versus alternatives like get_context or impact_radius. There are no exclusions or alternative tool references, so it falls short of providing clear usage guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_contextA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden and does an excellent job. It discloses the return format (source of top matches, file:line citations), and explains the effects of rerank=True and expand_graph=True, offering rich behavioral insight beyond the schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise at two sentences, front-loads the purpose, and then efficiently explains return behavior and flag effects. No filler or redundant information is present.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  An output schema exists, so return-value details need not be over-explained. The description covers purpose, key flags, and citation output, but omits k semantics and any usage prerequisites, leaving minor gaps for a no-annotation tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 0%, so the description must compensate. It explains the semantics of rerank and expand_graph, but leaves k and query largely implicit. With four parameters, the coverage is partial, not complete.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Retrieve' and identifies a distinct resource: a 'ready-to-cite context bundle' for answering a question. It mentions file:line citations and reranking, which clearly differentiates it from siblings like search_code or status.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly states the tool is for answering a question and producing grounded, citable spans, which gives clear context for when to use it. However, it does not explicitly name alternative tools or state when not to use it, so no exclusions are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• statusA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. The verb 'Report' strongly implies a read-only, non-destructive operation, but it does not explicitly state side-effect-free behavior. For a simple status tool, this is sufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, direct sentence that front-loads the purpose. No unnecessary words or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no parameters and an output schema available, the description fully covers the tool's purpose. The one-sentence description is complete for a status tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the baseline is 4. There is nothing for the description to add regarding parameter semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Report' and identifies the exact resources ('index size' and 'active embedding backend'). This clearly distinguishes it from sibling tools like 'search_code' or 'index_repo'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  While no explicit 'when to use' is stated, the description clearly implies the tool is for status checks. Given the tool name 'status' and the nature of the report, the context is clear enough, though no alternatives are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.