Server Quality Checklist

Tool Scores

• query_knowledge_hubB
  3.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses hybrid search (semantic + keyword) and that results include source citations. However, with no annotations, it omits details on idempotency, rate limits, or whether the search is stateful. Adequate but not comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Description is concise with four sentences, front-loading the core purpose. It lists parameters but could be better structured (e.g., using a list format). Still, it avoids unnecessary fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a search tool with 4 parameters and no output schema, the description covers the search mechanism and most parameters. However, the missing 'session_id' parameter explanation and lack of output format details leave gaps. Annotations would help, but none are provided.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Input schema has 100% coverage and descriptions, but the tool's own description omits the 'session_id' parameter entirely and provides only marginal rephrasing of schema descriptions for other parameters. It adds value by explaining hybrid search but fails to cover all parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states 'Search the knowledge base' with a specific verb and resource. It is distinct from sibling tools like delete_document and list_collections, establishing its purpose as a query/retrieval tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus siblings or alternatives. It does not mention conditions like 'use for document retrieval' or 'do not use for metadata-only queries.'

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ingest_documentA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses the internal process (parsed, chunked, embedded, stored) and supported file types. However, with no annotations, it should also cover potential side effects like overwrite behavior, authentication needs, or performance implications, which are omitted.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured: a one-sentence summary, followed by format support, processing steps, and parameter details. Every line provides unique information without repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the tool's function and parameters adequately. However, with no output schema, it does not describe the return value or confirm success. It also lacks details on error handling or duplicate handling, leaving some gaps for an AI agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, but the description adds clarifying details: file_path can be absolute or relative, and collection defaults to 'default'. These are valuable but modest additions, placing the score at the baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Ingest') and resource ('document file into the knowledge hub'), clearly stating the action and target. It also lists supported formats, which adds precision. The tool is distinct from siblings like delete_document or query_knowledge_hub.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance is given on when to use this tool versus alternatives. There is no mention of prerequisites, when not to use it, or references to sibling tools. The usage context is implied but not stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_document_summaryA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It discloses the return fields and implies a read-only operation, but does not cover error handling, auth requirements, or behavior on invalid input. The transparency is adequate but not exhaustive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise at 7 lines, uses bullet points for clarity, and front-loads the core purpose. Every sentence serves a purpose, and there is no redundant or filler content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simple nature (2 params, no output schema, no nested objects), the description provides sufficient context: what is returned and when to use it. It could mention error cases or edge conditions, but for its complexity level, it is reasonably complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with good descriptions for both doc_id and collection. The description adds value by explaining that doc_id accepts both full ID and hash, and collection defaults if omitted, but this information is already partially covered in schema. The description does not significantly deepen parameter understanding beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns summary and metadata for a specific document, listing specific fields (title, summary, tags, source path, chunk count). It distinguishes itself from sibling tools like delete_document or ingest_document by focusing on retrieval of structured information.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly advises using this tool after list_collections to get details about documents, providing a clear usage sequence. It does not specify when not to use it, but the context given is sufficient for a simple retrieval tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_collectionsA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses return information including conditional output for include_stats=true. No contradictions with missing annotations; covers behavioral traits adequately.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two short paragraphs, front-loaded with purpose, bullet points for structure, no redundant words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list tool with one optional parameter and no output schema, provides all necessary context: purpose, response shape, and usage hint.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage 100%, and the description adds value by explaining the effect of include_stats on output (document count conditional).

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states 'List all available document collections in the knowledge base', using specific verb and resource. Distinguished from siblings like query_knowledge_hub and delete_document.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says 'Use this tool to discover available collections before querying', providing clear when-to-use context. Does not explicitly mention when not to use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• delete_documentA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses destructive behavior by listing all removed data (vector embeddings, BM25 index entries, extracted images, ingestion history) and preview mechanism. No annotations, so description carries full burden and does so thoroughly.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two paragraphs with front-loaded purpose. Every sentence adds value; no redundancy or filler. Efficient and clear.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Comprehensive for a destructive tool: covers preview return details (chunk count, image count), what gets deleted, and the two-step workflow. No output schema, but description suffices.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema covers all parameters with descriptions. Description adds value by explaining the role of confirm_delete_data in preview vs execution, which goes beyond schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the tool deletes a document and associated data from RAG knowledge base, with a specific verb and resource. Distinguishes from sibling tools like ingest_document or get_document_summary.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit two-step process: first call for preview, second call to execute. However, does not explicitly state when not to use this tool or compare with alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.