Server Quality Checklist

Tool Scores

• docs_homeA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of behavioral disclosure, but it only states what the tool is, not what it does or its safety profile. It doesn't mention whether it's read-only, what output to expect, or any limitations, leaving the agent to infer safe behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise, front-loading 'Orientation' and using a second sentence for critical usage guidance. It wastes no words, though the terse style sacrifices some clarity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter, no-output-schema orientation tool, the description is minimally adequate. It tells the agent what the tool covers and to call it first, but doesn't specify the format or detail of the orientation content, which could lead to uncertainty.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the baseline is 4 per the rubric. The description correctly doesn't add parameter details, as there are none to explain.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies this as an orientation tool for registered llms.txt sources and how to search/fetch, which distinguishes it from sibling action-oriented tools. However, it lacks a strong verb (e.g., 'provides orientation') and is somewhat noun-phrase based.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The explicit instruction 'Call this first' provides clear when-to-use guidance as an entry point, and the mention of covering search/fetch implies its role relative to siblings. It doesn't explicitly name alternatives or exclusions, but the imperative is strong.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• refresh_doc_sourceA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry the full burden of behavioral disclosure. It communicates the action ('Re-index') and benefit ('pick up new/changed docs') but does not disclose potential side effects (like whether absent docs are removed), requirements (source existence), or whether this is a destructive operation. This is a significant gap for a mutation-like tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that clearly states the action and benefit. Every word earns its place; there is no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a one-parameter tool, the description is minimally adequate: it names the action, source of truth, and expected outcome. However, with no annotations or output schema, it lacks details on failure modes, idempotency, and whether the refresh is incremental or full. Given the low complexity, this is acceptable but not outstanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with parameter 'name' described as 'Source name to refresh', which aligns with the tool's description. The description adds no extra parameter semantics beyond what the schema already provides, so baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Re-index' with a clear resource 'a source from its llms.txt', and the parenthetical 'pick up new/changed docs' further clarifies the exact purpose. This distinguishes it well from sibling tools like add_doc_source, remove_doc_source, and search_docs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool (when new or changed docs need to be picked up from a source's llms.txt) but does not explicitly state when not to use it or mention alternatives (e.g., re-adding the source). Usage context is clear but exclusionary guidance is missing.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• remove_doc_sourceA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Since no annotations are provided, the description carries full burden. 'Remove' clearly indicates a deletion action, but it does not disclose whether the removal is reversible, if it cascades to related data, or if any confirmation is involved. This is acceptable for a simple operation but lacks extra context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence that immediately conveys the action. There is zero redundancy, and it is front-loaded with the verb. For a tool with one parameter, this level of conciseness is optimal.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple removal tool with a well-documented schema and no output schema, the description is sufficiently complete to convey core functionality. However, it lacks any note about side effects or post-conditions, which would make it more robust, especially as a mutation tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has one required parameter 'name' with the description 'Source name to remove', providing 100% coverage. The tool description adds no additional meaning to the parameter beyond what the schema already specifies, so it receives the baseline score for high coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's action (Remove) and target (a registered source), using a specific verb and resource pair. This distinguishes it from siblings like add_doc_source (adds), refresh_doc_source (refreshes), and list_doc_sources (lists). No ambiguity remains about the tool's purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when a source needs to be removed, but it does not explicitly state when to use this tool versus alternatives, nor does it mention any prerequisites or exclusions. The context of 'registered' hints that the source must already exist, but this is not elaborated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• add_doc_sourceA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses key behaviors: runtime registration, indexing, and persistence for future runs. However, it does not address potential side effects like duplicate name handling, idempotency, or error conditions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two short sentences that are front-loaded with the primary action and followed by the persistence behavior. Every word earns its place; no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given only two simple parameters and no output schema, the description is adequately complete for a registration tool. It explains what happens (indexing) and persistence across runs, which covers the main context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, as both `name` and `llms_txt_url` have descriptions. The tool description adds no further semantic detail beyond the schema, so the baseline of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the tool's action: 'Register a new llms.txt source at runtime and index it.' It uses a specific verb (register) and resource (llms.txt source), and the persistence note distinguishes it from sibling tools like remove_doc_source and refresh_doc_source.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The context is clear: this is for adding a new source at runtime, as opposed to listing, searching, fetching, removing, or refreshing. However, it does not explicitly mention when not to use it (e.g., for an existing source that needs update) or explicitly name alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_doc_sourcesA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of disclosing safety and behavior. 'List' implies non-destructive behavior, but no additional context is given about ordering, pagination, or whether sources are from a local or remote registry. The output fields provide some transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, focused sentence that front-loads the key information (action, object, output details) without any wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list tool with no parameters, the description covers the essential purpose and expected output fields. It lacks details like whether the list is exhaustive or if there are additional filters, but these are not critical for a basic listing operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool accepts zero parameters, so the baseline is 4. The description does not need to add parameter-level detail since there are none, and the schema is empty.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('List') and the resource ('registered documentation sources'), and specifies the output fields (name, llms.txt url, index status). This distinguishes it from sibling tools like add_doc_source or remove_doc_source, which perform mutations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage as a read-only listing tool, but it does not explicitly state when to use it instead of alternatives. Sibling names suggest it complements add/remove/refresh, but no direct guidance or exclusions are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• fetch_docA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden. It discloses that content is 'fetched live' and full content is returned, providing useful behavioral context. It doesn't explicitly state read-only or error behavior, but the fetch semantics are clear.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the main verb and resource. Every word earns its place: no filler, fluff, or repetition of the schema.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter tool with no output schema, the description fully covers what the tool does, how the input is constrained, and a key behavioral trait (live fetch). No further context is needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already documents the single parameter well (100% coverage), including its format and origin. The description adds a constraint ('must belong to a registered source') but does not significantly augment the schema's meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific action ('Fetch full content of a doc url') and the resource type. It distinguishes from sibling tools like search_docs, which search for docs, and list_doc_sources, which list sources.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly specifies the prerequisite 'use search_docs first' and the constraint that the url 'must belong to a registered source', providing clear when-to-use and when-not-to-use guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_docsA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden and delivers: it discloses the algorithm (BM25, Porter stemming, bigrams, markdown weighting), what the function returns (ranked {source,url,title,score,snippet}), and the scope ('registered llms.txt documentation'). It does not mention rate limits or auth, but for a search tool this is adequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, front-loaded with the core action, then usage guidance, then technical detail. Every sentence earns its place with no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a search tool with no output schema and no annotations, the description is remarkably complete: it specifies the return format, the follow-up action, the algorithm, the document scope, and the alternative approach. The agent can confidently select and invoke this tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds value beyond schema by giving concrete query examples, naming example sources, and outlining the follow-up with fetch_doc—reinforcing what each parameter means in practice.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('BM25 search') and clearly identifies the resource ('registered llms.txt documentation'), listing concrete examples (Strands, Kiro, AWS Bedrock). It distinguishes itself from siblings by framing the workflow as search_docs + fetch_doc, unlike list_doc_sources or fetch_doc.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states preference over per-product documentation MCP servers and provides the rationale ('lean, few round-trips'). Also implies when to use it versus fetch_doc by describing the two-step flow.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.