Server Quality Checklist

Tool Scores

• get_doc_sectionB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full responsibility for behavioral disclosure. It only says 'Retrieve' and provides no details on error handling, return format, permissions, or behavior for invalid URLs/identifiers. This is a minimal disclosure for a retrieval tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that communicates the core action and parameter without waste. It is appropriately sized for the tool's simplicity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has one parameter, no output schema, and no annotations, yet the description still fails to explain expected output or error behavior. Given the sibling context, there is no mention of when to use this tool vs. searching or summarizing, leaving the agent uncertain about edge cases.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% — the 'url' parameter is described as 'The URL or identifier of the documentation section'. The description adds no additional semantic detail beyond restating that the section is retrieved by URL or identifier, so a baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Retrieve') and resource ('a specific section of documentation'), clearly distinguishing it from siblings like search_docs or summarize_doc. The phrase 'by URL or identifier' adds scope, so the purpose is unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives no guidance on when to use this tool versus alternatives. It does not mention that search_docs should be used when the URL/identifier is unknown, nor any other exclusions or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_docsA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must carry the burden of behavioral disclosure. It mentions 'semantic search' and returns 'relevant documentation sections,' which gives some idea of behavior, but it doesn't disclose potential rate limits, authentication needs, whether results are snippets or full text, or whether any other side effects exist. It's minimal but not misleading.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, no wasted words. The first sentence is front-loaded with the action ('Search'), and the second explains the return type. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple 3-parameter search tool, the description is adequate but leaves gaps. It doesn't clarify the output format (e.g., URLs, titles, snippets) or how the optional source filter works. Given the sibling get_doc_section, it would help to state that search returns results that may be retrieved with get_doc_section. Still, the tool is simple enough that this is acceptable.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with parameter descriptions for query, limit, and source. The description adds no extra parameter semantics, so baseline 3 applies. It doesn't explain how the source filter interacts with the search or what the limit controls beyond the schema's description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool performs a semantic search over specific documentation sets (SUSE, Rancher, K3s) and returns relevant sections. This is a specific verb-resource pair that differentiates it from siblings like get_doc_section (retrieving a section) or summarize_doc (summarizing), even though it doesn't explicitly name them.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is given on when to use this tool versus alternatives. It doesn't mention exclusions like 'use get_doc_section for retrieving a known section' or 'use ask_question for natural-language questions.' The context implies it's for discovery, but sibling differentiation is absent.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• summarize_docA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions 'AI-powered' and 'open-source models' but does not disclose potential limitations, latency, or what happens if the URL is invalid or inaccessible. No side effects or prerequisites beyond the existence of a URL are stated.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-formed sentence that states the action and scope. It is front-loaded and contains no redundant or filler content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has a simple 2-parameter schema and no output schema, so the description should clarify expected return behavior. It does not explain what the summary output looks like or how it is returned (e.g., plain text, structured object). Given the absence of annotations and output schema, this is a notable gap, but the description covers the core purpose adequately for a simple tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%: both 'url' and 'format' have clear descriptions, including an enum for format. The tool description adds no additional parameter context, but with full schema coverage, the schema adequately documents the parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Generate' and clearly states the resource ('summary of a documentation page or section'), which unambiguously distinguishes it from siblings like get_doc_section or search_docs. It immediately communicates the tool's core function.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Usage context is implied by the description ('summary' vs. retrieval/search), but there is no explicit guidance on when to use this tool over siblings. For example, it does not state that this should be used instead of get_doc_section when a synthesized overview is needed or search_docs when looking for specific information.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ask_questionA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses that the tool searches docs and returns an answer with sources, but does not explicitly state that it is a read-only operation or describe limitations such as what happens if no relevant documentation is found. The behavior is partially transparent but lacks depth.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long, front-loaded with the core action, and contains no extraneous information. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with no output schema, the description adequately explains the result (an answer with sources). It is missing an explicit statement about how it handles out-of-scope questions, but overall it is complete enough for straightforward use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides complete descriptions for both parameters ('question' and 'context'). The description adds no additional semantic value for parameters, so the baseline of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: asking a question about SUSE documentation and returning an answer with sources. It distinguishes itself from siblings like search_docs (which likely returns raw search results) and summarize_doc (which summarizes content) by emphasizing AI-synthesized answers.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance is given on when to use this tool versus alternatives like search_docs or get_doc_section. The description implies it is for question-answering, but there is no mention of exclusions or when another tool would be more appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• index_documentationA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden for behavioral disclosure. It does not mention side effects, idempotency, or that indexing overwrites existing data, despite being a mutation tool. The presence of a 'forceRefresh' parameter hints at cache behavior, but the description omits any implications.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core action and purpose. Every sentence contributes value, and there is no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has two parameters, no output schema, and no annotations, the description is minimally viable but leaves gaps. It states the primary use case but does not explain the effect of indexing, the meaning of 'update the index,' or any prerequisites. This is adequate for a simple tool but lacks depth.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with both parameters documented. The description adds no extra meaning beyond the schema, so it meets the baseline but does not exceed it. The enum values for 'source' are not elaborated in the description, but that's acceptable given full schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's action ('Index documentation from specified sources') and its purpose ('for faster searching'). It distinguishes from sibling tools that are for retrieval/summarization, establishing this as a maintenance operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit timing guidance: 'run initially or to update the index.' This tells the agent when to invoke the tool, though it does not explicitly mention alternatives or when not to use it. Sibling tool names imply the distinction.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_doc_sourcesA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses that the tool returns status, but does not clarify whether the operation is read-only, how status is defined, or what happens if no sources exist. The behavior is mostly predictable for a list tool, but the description provides minimal extra context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no unnecessary words. It covers the essential action and result in a concise, well-structured manner.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple, zero-parameter list tool, the description is largely complete. It explains the purpose and that status is included, but does not detail the exact return format or the meaning of 'status'. Given the tool's simplicity and the context from sibling names, this is adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so there is nothing for the description to explain. Per the rubric, a baseline of 4 is appropriate when no parameters exist, since the schema covers everything vacuously.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's action ('List'), resource ('documentation sources'), and additional detail ('and their status'). It is easily distinguished from sibling tools that get sections, search, summarize, ask, or index documentation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Usage is implied rather than explicitly stated: it lists available sources, so one would use it to get an overview. However, there is no explicit guidance about when to choose this over alternatives or any exclusions, leaving the agent to infer.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.