Server Quality Checklist

Tool Scores

• extract_code_snippetsB
  3.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It clarifies that the tool returns an inventory and lists the included fields such as doc key, title, heading, language, line number, source URL, and code. It does not address filtering behavior, output limits, or empty-result handling.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded with the core purpose. The result-field list and the use-case sentence add useful context without unnecessary repetition or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has three optional parameters, no annotations, and an output schema, yet the description leaves parameter semantics unexplained and does not guide selection versus sibling tools. The purpose and result contents are covered, but an agent still lacks critical information needed to invoke the tool effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description does not explain any of the three parameters: key, language, or max_snippets. The word 'language' appears only in the result-field list, which does not help an agent understand how to populate the language parameter correctly.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Return an inventory') and a distinct resource ('fenced code snippets from the documentation'), making the tool's main function clear. It does not explicitly contrast with sibling tools such as find_examples, but the fenced-snippet inventory scope is sufficiently distinct.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description names concrete use cases: 'snippet verification and docs CI design.' However, it does not state when not to use this tool or name alternatives like find_examples, so it provides clear context but no exclusionary guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_examplesA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must convey behavioral traits on its own. It only says what the tool finds, not what happens when called (return format, filtering, pagination, safety). For a search-type tool this leaves the agent without crucial insight into the response shape.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences: the first states the core purpose, the second provides clear use-case guidance. No filler or redundant phrasing; the structure is efficient and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is relatively simple, and there is an output schema (though not shown). The description gives purpose and usage context, but with zero annotations and no parameter detail, it leaves gaps about result limits, error behavior, and how language relates to query. It is adequate but lacks full coverage for a 3-parameter tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It mentions 'topic and language', loosely mapping to query and language, but gives no detail on query syntax, accepted language values, or the meaning and effect of max_results. The parameter semantics remain largely inferable only from parameter names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Find'), a resource ('code examples in the Iggy documentation'), and the two key dimensions ('topic and language'). This clearly distinguishes it from sibling tools like list_docs or get_doc, which are not example-specific.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly lists when to use this tool: runnable examples, SDK examples, producer/consumer samples, Docker commands, or tutorial source material. However, it does not mention when not to use it or suggest alternatives, so it stops short of the full 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_docA
  3.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It only says 'retrieve', implying a read-only operation, but it does not mention error behavior, rate limits, or what happens when the key is invalid. This is a low-information description for a tool with no annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no wasted words: the first names the operation, and the second gives immediate pointer to list_docs. It is well-structured and easy to scan, though it could include a note about the return format without bloating the description.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter retrieval tool with an output schema, this is adequate. The description tells you exactly how to get a valid key and what the tool returns (full text). However, it lacks any mention of failure modes or what happens when the key is missing, and the output schema is not described in the description itself.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides only the key name and type with 0% comprehensive coverage. The description compensates by giving concrete examples of valid key values ('server/docker', 'sdk/python/intro') and points to list_docs for discovering keys, adding meaningful usage context beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Retrieve') and a specific resource ('the full text of one Iggy documentation page'), which clearly distinguishes this tool from sibling tools like search_docs or extract_code_snippets. It also provides concrete example keys, making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly tells the agent to use list_docs to discover valid keys, providing a clear prerequisite and usage sequence. It does not explicitly contrast with other siblings like search_docs, but it is strongly implied that this tool is for retrieving a single document while list_docs is for discovery.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_related_docsA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It adds that 'Relatedness comes from explicit links, backlinks, shared section, and shared vocabulary', which explains the tool's internal logic, but it never explicitly states this is read-only or discloses ordering/rate-limit behavior. The verb 'Return' implies no side effects, but it isn't confirmed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences with no wasted words: purpose first, then the relatedness algorithm, then a usage tip. Information is front-loaded and easy to scan.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description supplies purpose, algorithm, and typical usage context; an output schema exists so return details are covered. The only notable gap is that max_results is undocumented in the description, which prevents fully anticipating the response shape or pagination behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It clarifies that key means 'doc key', but max_results is never mentioned or explained beyond the parameter name and default. An agent cannot learn from the description how the max_results parameter affects the call.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Opens with 'Return documentation pages related to one doc key' – a specific verb, object, and scope. This differentiates it from get_doc (single page), search_docs (query over docs), and list_docs (catalog), which are all siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states 'Use this after get_doc when one page may not contain all context needed for a complete answer', giving a specific predecessor and a condition. It doesn't name other alternatives or say when not to use this tool, but the context is clear enough for an agent to choose correctly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_docsA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It openly explains that results are labeled by source, notes that guides and Rust API docs may disagree, and describes the effect of the 'source' parameter. This goes beyond a simple 'search' statement and gives useful behavioral expectations, though it stops short of describing output format or error behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded with the core action. Each subsequent sentence adds valuable context: the two-corpus behavior, source filtering, and the strong recommendation to use this tool over memory. Nothing is redundant or wasted.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity and the existence of an output schema, the description is quite complete. It explains what the tool does, how to narrow the search, and why it matters. The only minor gap is the lack of mention of 'max_results' or result ordering, but the output schema likely covers result structure, so this is a minor shortfall.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It explicitly explains the 'source' parameter and implicitly covers 'query' via 'search for a term or phrase', but it gives no explanation for 'max_results'. Since only two of the three parameters are addressed, the compensation is partial, warranting a middle score.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource ('Search the Apache Iggy documentation') and clearly distinguishes its scope (searches both hand-written guides and Rust doc comments) from sibling tools like list_docs and get_doc. An agent can immediately understand what this tool does and what makes it unique.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear usage context: it explicitly tells the agent to prefer searching docs over answering from memory, and explains when to use the 'source' parameter to narrow the search. However, it does not explicitly compare against sibling tools or state when not to use them, though this is implicitly clear from their names.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• explain_doc_mapA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It explains the tool's nature ('guided map, not a keyword search'), implies it works over the indexed corpus, and indicates the kind of output (an explanation, not raw results). This is adequate for a read-only explanation tool, though it could mention limitations or fallback behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long, with the primary action front-loaded in the first sentence and usage guidance in the second. Every sentence contributes meaning, and there is no fluff or repetition. It is efficiently structured and easy to scan.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers what the tool does, when to use it, and its behavior in relation to sibling tools. With an output schema present, the return structure is already defined. The only minor gap is that it doesn't explicitly describe the expected param values or examples, but given the simple optional params and clear intent, the context is sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema provides no descriptions for the two parameters, so the description must compensate. It indirectly explains the 'goal' parameter by referencing 'what reading path fits a goal,' but it never explicitly states what values 'goal' or 'audience' should take. The parameter names and defaults are self-explanatory, so the description adds moderate but not full semantic value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a clear verb and resource: 'Explain how the Iggy documentation is organised and where to start.' It immediately distinguishes itself from a keyword search by labeling itself 'a guided map,' which differentiates it from sibling tools like search_docs. The scope is precise and leaves no ambiguity about what the tool does.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit usage conditions: 'Use it when a user asks where docs live, what reading path fits a goal, or which documentation areas look thin from the indexed corpus.' It also states a negative condition: 'not a keyword search,' which signals when not to use it and points toward search_docs as the alternative. This is strong routing guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_broken_linksA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description covers the essential behavioral traits: it states internal links are validated against the indexed corpus and heading anchors, while external URLs are merely counted unless check_external is enabled. This gives the agent a good mental model of what the tool does. It doesn't explicitly mention side effects (e.g., read-only), but a link-checking tool is naturally non-destructive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no filler. The first sentence states the tool's primary purpose, and the second explains the crucial external-link behavior. Every word earns its place, and the information is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with a single optional parameter and an output schema, the description provides enough context to invoke it correctly. It explains how internal and external links are handled and the role of the flag. Minor gaps remain (e.g., what 'counted' means exactly, or how external checks are performed), but these do not block correct usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% description coverage, so the description must carry the meaning of 'check_external'. It does exactly that: 'External URLs are counted but skipped unless check_external is true.' This clarifies that setting the flag to true switches external URL handling from skip to active checking, adding the needed semantics beyond the bare parameter name.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('find broken links') and the specific target ('local Iggy documentation checkout'). It also distinguishes internal links (checked against corpus and anchors) from external links (skipped by default), which sets it apart from sibling tools like list_docs or search_docs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains the default behavior for external links and the condition under which they are checked ('unless check_external is true'), giving the agent clear context for when to set that parameter. It does not explicitly list alternatives or when-not-to-use cases, but the scope is narrowly defined enough for this tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_docsA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  There are no annotations, so the description carries the burden. It discloses that this is a comprehensive listing operation ('List every'), how the output is organized by section, and that passing the source parameter filters to one half. It also explains the provenance of the webpage and Rust API docs, which is useful behavioral context. The description does not mention side effects because it is inherently read-only, and that is reasonably inferred from the verb 'list.'

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise: one clear first sentence and one short paragraph providing the corpus organization details. Every sentence adds meaningful information about what the tool lists and what the source parameter controls. No filler or redundant phrasing is present.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With one optional parameter and no enums, the description covers the tool's purpose, the parameter semantics, and the default behavior. The presence of an output schema means return value details need not be documented in text. The only minor gap is that the set of accepted source values is not given as an explicit enumerated list, but it is clearly inferable from the prose, so overall the context is complete enough for an agent to call the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides only a string parameter named 'source' with no description or enum, and schema description coverage is 0%. The description fully compensates by naming the two possible halves ('website' and 'rust-api'), explaining that the default combines both, and instructing 'Pass source to see one half alone.' This gives the agent concrete parameter values and semantics beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states the tool's function: 'List every Apache Iggy documentation page, organised by section.' This is a specific verb and resource, and the detail about the two corpus halves ('website' and 'rust-api') clarifies the scope. The strong contrast with sibling tools like get_doc or search_docs is evident, so an agent can distinguish it without looking at others.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains how to use the tool, especially the source parameter and that the default lists all pages. It gives clear context about the two subsets of the corpus, but it does not explicitly name alternative tools or state when not to use this one. Still, the guidance is sufficient for a listing operation where no heavy alternative-picking logic is required.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• refresh_docsA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full responsibility. It reveals two important behaviors: bypassing the cache and being a no-op under specific conditions. It stops short of noting potential side effects like network latency or failure modes, but for a cache refresh this is adequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, each earning its place: the first states the primary function, the second clarifies an important conditional exception. No redundancy or extraneous detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a parameterless operation with an output schema present, the description fully covers what the tool does, when to use it, and the local-checkout caveat. The agent has everything required to decide whether to invoke it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters and the schema is empty, so the baseline is 4. The description adds no parameter semantics because none exist; nothing further is needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states a specific verb ('Re-fetch') and resource ('documentation corpus'), and explains the key behavior ('bypassing the seven-day cache'). This distinguishes it from sibling read-only tools like list_docs or get_doc, which operate on the current corpus.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly describes when to use the tool (when a cached remote copy needs refreshing) and when not to use it (when IGGY_DOCS_ROOT points to a local checkout), and names the alternative action ('run git pull there instead'). This is strong practical guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.