Server Quality Checklist

Tool Scores

• search_docsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of disclosing behavior. It explains the search behavior and output contents, including similarity scores and source files, but does not explicitly state side effects, read-only guarantees, or error handling. This leaves some behavioral details unstated.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well structured. It states the main purpose, summarizes the return content, and provides the key conditional usage note without unnecessary detail or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is mostly complete for a semantic search tool: it covers the query, collection behavior, and result content. However, the lack of explanation for top_k leaves a meaningful gap, since an agent cannot confidently know how many results to expect without inferring it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema provides no parameter descriptions, so the description must compensate. It explains the query and collection parameters reasonably, including multi-collection behavior, but does not explain top_k or its default meaning. This is a notable gap in parameter semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool performs semantic search over indexed documents and returns relevant passages with similarity scores and source files. It is distinct from the sibling listing tools, which focus on enumerating sources and collections rather than searching content.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives practical guidance for multi-collection mode, telling users to specify a collection or leave it empty to search all collections. It does not explicitly name sibling tools as alternatives, but the conditional usage instruction is clear enough for typical search scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_collectionsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It implies a read-only listing operation and mentions the availability condition, but it does not explicitly state that it has no side effects, nor does it describe error behavior or output format details beyond 'chunk and file counts'.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise, consisting of two short sentences. It front-loads the primary purpose and then adds a necessary condition. There is no fluff or redundant information, making it highly efficient for an agent to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with no parameters, the description is nearly complete. It states what is returned (chunk and file counts) and when it is available. It lacks explicit notes on edge cases or errors, but for a listing operation, the provided context is sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and the schema coverage is trivially 100%. According to the rubric, the baseline for high coverage is 3, and there is no parameter information to add. The description does not need to explain parameters, but it also does not enhance anything beyond the baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: listing available collections and providing chunk and file counts. It uses a specific verb ('List') and object ('collections'), making the purpose unambiguous even without comparing to siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides an explicit precondition for use: 'Only available in multi-collection mode (LORE_DB_DIR set)'. This guides the agent on when this tool is applicable. However, it does not contrast with alternatives like search_docs or list_indexed_sources, so the guidance is not fully comprehensive.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_indexed_sourcesA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. 'List' clearly implies a read-only operation, and the description states what is returned (indexed files with chunk counts). It does not mention edge cases like pagination or errors, but for a simple listing tool this is reasonably transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences with no filler or redundancy. The core purpose is front-loaded, and the parameter behavior is explained efficiently in the second sentence.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the low complexity (one optional parameter) and the presence of an output schema, the description covers the essential usage scenarios. It could be slightly more explicit about behavior in single-collection mode, but overall it is sufficient for an agent to call the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Although the input schema has 0% description coverage, the tool description directly explains the only parameter: 'specify a collection name or leave empty to list sources across all collections.' This fully clarifies the meaning and default behavior of the collection parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action ('List'), a specific resource ('indexed files'), and the output ('with chunk counts'). It is clearly distinct from sibling tools like search_docs and list_collections by describing an inventory-style listing.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains the conditional behavior of the collection parameter ('In multi-collection mode... leave empty to list sources across all collections'), but it does not explicitly compare this tool with sibling tools or state when to prefer this over search_docs or list_collections. Usage is implied rather than stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.