Server Quality Checklist

Tool Scores

• runbook_addA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must carry the full burden of disclosure. It clearly indicates a write operation ('Record') but does not mention whether entries can duplicate, whether it overwrites, or any side effects. Lacks details on idempotency or persistence.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single, clear sentence that is front-loaded with the action and purpose. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple add operation with fully documented parameters, the description is mostly adequate. However, it lacks context about when to use it versus runbook_search and does not mention the optional tags or symptoms parameters. No output schema means return behavior is undisclosed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description mentions 'symptom and its fix', which aligns with the parameters, but adds no additional meaning beyond the schema's own descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'Record' and identifies the resource ('a new symptom and its fix' in 'the local runbook'). It clearly distinguishes from siblings like runbook_search by indicating an add operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when someone has discovered a symptom and fix to preserve, but it does not explicitly differentiate from alternatives or state when not to use. No exclusion or comparison to runbook_search is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_endpointA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden. It discloses the core behavior (making a request, checking status, measuring time) and adds the 'answering vs running' context, but omits details like HTTP method, redirect handling, or timeout error behavior, leaving gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the action and outcome. No unnecessary words; every phrase earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple HTTP-check tool with fully described parameters and no output schema, the description adequately explains purpose, input, and return meaning. It lacks explicit return format details but is sufficient for this complexity level.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description mentions 'expected status code' and timing, which aligns with expectStatus and timeoutMs, but does not add additional meaning beyond what the schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action ('Request a URL') and the output ('report whether it returned the expected status code, plus how long it took'), clearly distinguishing it from sibling tools like tail_log and runbook_search.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear context for use ('Use it to confirm a service is actually answering, not just running'), which implies when to choose this tool, though it does not explicitly mention when not to use it or alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• runbook_searchA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It implies a read-only search operation but does not explicitly state whether the tool modifies anything, what it returns, or any performance characteristics. This is adequate but minimal for a search tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise, using two short sentences to convey purpose and usage. It front-loads the core action and immediately adds practical advice, with no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema, the description cannot rely on structured return types. It provides enough context for a simple search tool—what it searches, why to use it, and a high-level example—but lacks explicit detail about the result format or edge cases. This is sufficient for a low-complexity tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, with both 'query' and 'limit' already described in the schema. The description adds no additional parameter-level meaning beyond what the schema provides, so it meets the baseline without enhancing understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's verb ('Search') and resource ('the local runbook of known symptoms and their fixes'). It distinguishes itself from sibling tools like tail_log and check_endpoint by focusing on knowledge lookup rather than live system interaction, making its purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly recommends using this tool before diagnosing an issue from scratch, providing clear guidance on when to invoke it. It does not name alternatives or exclusion scenarios, but the 'before diagnosing' context implies when it should be preferred over other investigation methods.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• tail_logA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of disclosing behavioral traits. It explicitly says 'Read' (implying a safe, non-mutating operation) and mentions the 'configured log root' and case-insensitive filtering. However, it does not describe error behavior, return format, or what happens if the path is invalid, which are important for a tool with no structured metadata.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence that front-loads the primary behavior and includes optional parameters. Every word adds meaning, with no fluff or repeated information from the schema.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity, the rich schema descriptions, and the absence of an output schema, the description provides sufficient context for core usage. It does not specify the return format (e.g., array of strings vs. single string), but this is a minor gap for a straightforward log tail operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with clear descriptions for all three parameters. The tool description adds extra value by noting that the 'contains' filter is case-insensitive and that the path is relative to the configured log root, which is not evident from the schema alone. This goes beyond the baseline for well-documented schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool reads the last N lines of a log file under a configured root, with optional case-insensitive substring filtering. This is a specific verb ('Read') and resource ('log file'), and it is distinct from sibling tools like check_endpoint or runbook_search.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool—whenever recent log output is needed. It clearly defines the scope (log root, last N lines) but does not explicitly discuss when not to use it or mention alternatives, though siblings are unrelated and no exclusion is necessary.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.