Server Quality Checklist

Tool Scores

• kb_reindexB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavioral traits itself. It only says 'Re-run an indexer' and does not mention side effects, runtime, permissions, idempotency, or potential impact, leaving the agent with little safety context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely lean: one action sentence plus one parameter explanation. Every element carries information, and the core purpose is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a one-parameter tool with an output schema, the core invocation details are present. However, with zero annotations, the absence of any behavioral context such as side effects or prerequisites leaves moderate gaps for an agent deciding whether and how to invoke it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema only defines source as a string, and schema description coverage is 0%. The description compensates by explaining that source is an indexer name and tells the caller to obtain it from kb_list_sources, which is meaningful guidance.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a clear verb and resource: 'Re-run an indexer.' This is distinct from the sibling tools kb_search, kb_get, and kb_list_sources, though it does not elaborate on what re-running entails.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It implies the intended workflow by directing the caller to use an indexer name from kb_list_sources, but it does not explicitly state when to use this tool over alternatives or when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• kb_list_sourcesA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must carry the behavioral disclosure burden. 'List' implies a read-only operation and 'state of their Qdrant collections' indicates the kind of information returned. However, it does not explicitly state that it is non-mutating, requires no setup, or has no side effects. It is minimally adequate but not rich.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, direct sentence that efficiently communicates the tool's purpose. Every word earns its place, and the core action is front-loaded with 'List configured sources.' No filler or redundant phrasing is present.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has zero parameters and an output schema is present, the description does not need to explain return values. It captures the essential scope ('configured sources' and collection state). It could be more complete by noting how this relates to sibling tools, such as using it before reindexing, but overall it is sufficient for a simple list operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so schema coverage is trivially 100%. The baseline for 0-parameter tools is 4. The description does not add parameter-specific meaning, but none is needed because there are no parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb ('List'), a specific resource ('configured sources'), and the additional detail of 'the state of their Qdrant collections.' This distinguishes it from the sibling tools (kb_search, kb_get, kb_reindex), which imply searching, retrieving, and reindexing rather than listing sources.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives no explicit guidance on when to use this tool versus the siblings. It does not mention scenarios like checking configured sources, verifying collection state before reindexing, or troubleshooting. Usage must be inferred from the word 'List,' so this is a clear gap.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• kb_searchA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description bears the burden of explaining behavior. It adds non-obvious details: omitting source searches every configured collection, and top_k is per collection. It does not mention read-only status, auth, or rate limits, but search inherently implies a non-mutating operation and the output schema covers result structure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and well-structured: a brief one-line summary followed by a clean Args block. Every sentence and bullet adds necessary semantic value, with no filler or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description fully covers invocation details for all parameters and references kb_list_sources for source names. The existence of an output schema means return values do not need to be described. The only minor gap is the lack of explicit guidance on choosing this tool versus kb_get or kb_reindex, but this does not prevent correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, yet the tool description fully explains all three parameters. It clarifies that query is a natural-language question or keyword phrase, source is optional and filterable via kb_list_sources, and top_k limits hits per collection. This is excellent compensation for the schema's lack of descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Semantic search across indexed knowledge,' which clearly names the operation (semantic search) and the resource (indexed knowledge). It readily distinguishes this tool from siblings like kb_get, kb_list_sources, and kb_reindex, which naturally serve different operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives useful parameter-level guidance, such as using an optional source and referencing kb_list_sources, but it does not explicitly state when to prefer kb_search over kb_get or kb_reindex. The intended usage is implied by the name and opening sentence, but no exclusions or alternative tool routing are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• kb_getA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full behavioral disclosure burden. It explains the result content ('full text and payload') and adds important scoping behavior: when source is omitted, every known collection is checked. It doesn't cover not-found behavior or auth requirements, but for a simple getter this is reasonably transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded: one clear summary sentence followed by a concise Args section. Every sentence contributes meaningful information with no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a two-parameter getter with an output schema present, the description covers the essential call semantics and parameter behavior. It could have explicitly pointed to kb_list_sources for valid source values or described not-found behavior, but these are minor gaps given the tool's simplicity and the availability of an output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description fully compensates by explaining both parameters. It defines 'id' as the chunk id returned by kb_search, and 'source' as an optional collection-scoping hint with explicit default behavior when omitted. This adds real meaning beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Fetch the full text and payload of a chunk by id.' This clearly distinguishes the tool from siblings like kb_search, kb_list_sources, and kb_reindex, which serve different operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear context: the id should come from 'kb_search', and the source hint scopes the lookup, with the default behavior of checking every known collection if omitted. It doesn't explicitly state when not to use this tool or name alternatives, but the linkage to kb_search provides strong usage guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.