Server Quality Checklist

Tool Scores

• search_documentsB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, and the description does not disclose behaviors such as read-only status, authorization needs, or side effects. It only states the search functionality.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with a clear purpose sentence followed by parameter list. It is front-loaded and contains no unnecessary text, though could be slightly more structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The output schema exists, so return values need not be covered. However, missing details like maximum top_k, ordering of results, or clarification of 'hybrid' method make it adequate but not complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Despite 0% schema description coverage, the description explains all three parameters (query, top_k, method) with clear semantics, going beyond the schema's titles and defaults.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches the knowledge base for passages relevant to a query, distinguishing it from siblings like answer_question (answering) and get_stats (statistics).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives (e.g., answer_question). The description only lists parameters without context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• answer_questionA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must fully disclose behavior. It mentions answering with citations but does not specify output format, error handling (e.g., no answer found), authentication requirements, or whether it's read-only. This is insufficient for a tool with no annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is short and front-loaded, stating the purpose in the first sentence. The args section is clear but repetitive. No extraneous words. Could be slightly more structured, but overall efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema, the description should explain return values (e.g., format of citations). It mentions citations but not their structure. For a simple tool with 2 parameters, it is adequate but not fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must add meaning. It explains 'question: the user's question' and 'top_k: how many passages to ground the answer in,' which adds value beyond the schema names but remains minimal. It does not elaborate on valid values or constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Answer a question grounded in the knowledge base, with citations.' It specifies the verb (answer), resource (question grounded in knowledge base), and distinguishes from sibling tools like search_documents (document search) and get_stats (statistics).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for fact-based questions requiring citations, but does not explicitly state when not to use or suggest alternatives. The context is clear, but explicit when-not-to-use guidance is missing.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_statsA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden for behavioral disclosure, but it only states what is returned, not whether the tool is read-only, requires authentication, or any side effects. This is insufficient for a tool with no safety annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence that directly conveys the tool's purpose without any extraneous words, perfectly front-loaded and efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description could provide more details about return format or whether stats are cached/real-time. It covers the basic purpose but lacks completeness for a rich contextual understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters exist, so schema coverage is 100% by default. The description adds no parameter-specific info, but as baseline for zero parameters, it is adequate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the tool returns index size and active backends, distinguishing it from sibling tools like answer_question or search_documents which serve different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit when-to-use or when-not-to-use guidance is provided. The description implies usage context (retrieving stats) but lacks exclusions or alternatives, which is adequate but not helpful.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_sourcesA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description carries full burden but only states the basic action; it does not disclose read-only nature, authorization needs, or output format, though the output schema may compensate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence of 8 words, perfectly concise and front-loaded with the core action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given zero parameters and an existing output schema, the description is fully complete for this simple listing tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Zero parameters, so baseline is 4; no additional parameter info needed as schema coverage is 100%.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'list' and resource 'documents available in the knowledge base', clearly distinguishing it from sibling tools (answer_question, get_stats, search_documents) by focusing on enumeration of all sources.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool vs alternatives; usage is implied by the purpose of listing all documents, but no when-not or exclusion criteria are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.