Server Quality Checklist

Tool Scores

• delete_collectionA
  3.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description explicitly notes the destructive nature ('Delete', 'Irreversible') and the cascading effect on chunks. Without annotations, this disclosure is sufficient for understanding the tool's impact.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise with two sentences. The first sentence states the action, and the second adds critical irreversibility warning. No redundant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple delete tool with one parameter and an existing output schema, the description provides the essential behavioral details. It covers what is deleted and the permanence, though it could mention return value briefly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The single parameter 'collection' is not described beyond the schema. With 0% schema coverage, the description should clarify what the collection identifier represents, but it adds no semantic value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'delete' and the resource 'collection', and mentions 'all of its chunks', which distinguishes it from sibling tools like list_collections and ingest_text.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It only states the action and irreversibility, but lacks explicit when-to-use or when-not-to-use context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• searchA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must cover behavioral traits. It mentions semantic search and returns chunks with scores, indicating a read operation. However, it does not disclose permissions, rate limits, or whether it is read-only (though likely). The description is adequate but lacks depth.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise with two sentences containing no unnecessary words. It efficiently conveys the core functionality and output format.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the output schema exists, return values are covered externally. However, with 0% schema description coverage and no parameter explanations, the description is incomplete. It does not clarify the meaning of 'chunks', sorting order, or how the score is used, leaving gaps for an AI agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description should explain parameters. It does not describe 'k', 'query', or 'collection' beyond their presence in the schema. 'query' and 'collection' are self-explanatory but could benefit from format hints, while 'k' default of 5 is not justified. The description adds minimal value over the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool performs semantic search on a collection and returns top-k chunks with a cosine score. The verb 'search' and resource 'collection' are specific, and it distinguishes from siblings like delete_collection, ingest_text, and list_collections, which manage collections rather than search them.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for semantic search, but it does not explicitly state when to use this tool versus alternatives. Since it is the only search tool among siblings, usage is clear, but there is no guidance on when not to use it or any prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ingest_textA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, and the description only reveals that the tool modifies data and uses chunking/embedding. It does not disclose side effects (e.g., overwriting, idempotency), auth requirements, or error conditions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise: two sentences for the action and two lines for parameters. No unnecessary words, and the action is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool performs a complex pipeline (chunk, embed, store), but the description omits critical details: success return value, collection creation behavior, chunking parameters, and embedding model. Given no annotations, the description is insufficient for safe invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Parameter descriptions exist only for 'collection' and 'metadata' in the description, providing semantic context. 'text' is unexplained, and output schema is not addressed. With 0% schema coverage, this adds some value but is incomplete.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's action: chunk, embed, and store text for semantic search. It distinguishes from siblings (delete_collection, list_collections, search) by being the ingestion tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for adding text to a collection for later retrieval, but does not explicitly state when to use or avoid it, nor mention prerequisites like collection existence.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_collectionsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It indicates a read operation (list all collections) but does not explicitly state it is non-destructive, nor does it mention any other behavioral aspects like rate limits or pagination behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence with no unnecessary words. Every part of the sentence is informative.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no parameters and an output schema exists (though not shown), the description is mostly adequate. However, it could provide more context about what a collection is or any prerequisites, but for a simple listing tool, it is sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the schema provides full coverage. Per guidelines, baseline score for 0 parameters is 4. The description adds no param info, which is acceptable.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'List' and the resource 'collections' and specifies that chunk counts are included. It distinguishes itself from siblings (delete, ingest, search) by being a read-only listing operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives like search or ingest_text. The description does not provide any context for appropriate usage scenarios or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.