Server Quality Checklist

Tool Scores

• toolC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Despite the annotation destructiveHint=true, the description makes no mention of potential side effects or modifications to the environment. It only references error handling ('a tool this machine does not have says so'), omitting any disclosure about destructive behavior, which is a significant transparency gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is verbose and meandering, using awkward metaphors like 'a 4,000-line structural search costs a screenful' and redundant phrasing. Its single-paragraph structure lacks clear organization, making it harder to parse the essential information quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The output format is only vaguely described as 'a distilled view of what it printed', without concrete details on structure or content. It mentions that output is 'reachable through peek' but does not explain the full response shape or error scenarios beyond a cryptic reference to unavailable tools, leaving gaps for an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema includes two parameters (name and args) with zero description coverage. The description does not explain what 'name' refers to (presumably the tool name among sg/diff/loc) or how 'args' should be structured, leaving the agent without any guidance on how to populate these fields.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool runs one of three specific tools (sg, diff, loc) and returns a distilled view, which distinguishes it from generic tools. The phrase 'reach for sg instead of reading candidates' and similar clarify the intended use cases, though the description is convoluted and could be more direct.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides guidance for when to use the underlying tools (sg for search, loc for sizing) and mentions distillation of large output. However, it does not explicitly compare this tool to sibling tools like 'run' or state when to prefer this over alternatives, leaving the choice partially implicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• digestB
  3.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the readOnly/idempotent annotations, the description says the server opens the file, output is limited to roughly a screenful, and any included line is copied byte-for-byte rather than paraphrased. This is useful behavioral context, though it does not cover errors or permission failures.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the purpose and includes necessary usage boundaries, but the closing sentence is long and convoluted, mixing output characteristics with a peek comparison. It could be tightened without losing meaning.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  It gives good context about when to use the tool and what output behavior to expect, and an output schema exists, but the unexplained optional parameters and lack of error/edge-case guidance leave the description incomplete for full API usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has no parameter descriptions and the description explains only the path/file concept. The optional keep and budget parameters are never defined, so callers cannot know their accepted values or effect.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly says the tool reads a file and returns a distilled view of its contents, with examples of file types. It distinguishes this from outline and peek, though the phrase 'distilled view' is slightly imprecise.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It says to use digest for large textual files that would flood the conversation and explicitly says to use outline instead for source-code declarations. It also contrasts with peek for full-range reading, but does not mention digest_many for multi-file cases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• peekA
  3.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description goes beyond the readOnly/idempotent annotations by explaining behavior such as range clamping, default return of whole content when no range is specified, and grep with context lines. These details are valuable for understanding side-effect-free behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is moderately concise but contains poetic redundancies like 'open up a gap that a view left behind' and 'the way into a gap when you know the word you want'. These add style but not substance, making it less crisp than necessary.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers core behaviors and parameter basics, but the unexplained 'cap' parameter leaves a gap in the overall picture. Since no output schema is provided, the description is the only source of information about what the tool returns, and it is mostly sufficient but not fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description explains handle, range (first/last), around (context), and grep. However, the 'cap' parameter is not mentioned at all, leaving its purpose ambiguous. Schema description coverage is 0%, so this omission is meaningful.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns exact original lines of a capture or file with line numbers, which is a specific and distinct purpose. However, the flowery phrasing ('open up a gap that a view left behind') slightly obscures the directness.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides some usage context (use with handle from run result or any path) but does not explicitly compare to sibling tools like outline or digest. The guidance is implied rather than explicit about when to prefer this tool over others.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• digest_manyA
  3.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the annotations (readOnly, openWorld, idempotent), the description discloses additional behavior: per-file output ('Each file keeps its own last line') and partial failure handling ('a file that cannot be read says so in its place rather than taking the others down'). This enriches the annotation-only context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is somewhat wordy and repetitive, e.g., 'asked at the same time' and 'asked together' convey the same idea. It could be more concise while retaining the essential points.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  While the description covers high-level batching and error handling, it omits important context such as parameter meanings, output format details beyond a per-file line, and how 'budget' or 'keep' affect behavior. This leaves agents under-informed given the tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description provides no explanation of the parameters ('paths', 'keep', 'budget'). Schema coverage is 0%, and the description does not compensate; it only vaguely references file lists without naming or defining any parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Digest several files in one call' with the verb 'digest' and resource 'files'. It differentiates from single-file alternatives by emphasizing batching, e.g., 'come back as one tool result instead of four'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit usage condition: 'Use it whenever there is more than one file to read'. It also contrasts with multiple individual calls ('four logs cost four waits... roughly one wait') and describes error behavior for unreadable files, providing clear guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• outlineA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already cover readOnlyHint, idempotentHint, and openWorldHint. The description adds valuable behavioral details: 'it holds no table of languages and never reads the suffix', 'Lines come from the file byte for byte', and the comparison with `peek`. These go beyond the annotations and clarify the tool's universal applicability and reading behavior, though it does not mention auth or side effects, which are not needed for a read-only tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded with the core purpose. Each sentence adds value: the first explains what it returns, the second clarifies its universal applicability and relationship to `run`, and the third gives a behavioral detail about reading and contrasts with `peek`. There is no wasted text; it is efficient and information-dense.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  While the purpose and behavior are thoroughly explained, the description lacks any guidance on the two optional parameters (`keep` and `budget`). Since the schema provides no descriptions for them, an agent cannot know what values to pass or how they affect the output. The output schema exists, but the description should still give enough context for correct invocation, which it does not.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate for explaining the parameters. It does not explain `keep` or `budget` at all, and only implies `path` by context. The description focuses on the tool's behavior but leaves the optional parameters unexplained, which is a significant gap for an agent trying to call the tool with appropriate arguments.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states the tool's function: 'Return what a file declares -- its types, functions, exports, targets and settings -- without their bodies.' It also clearly distinguishes it from siblings by explaining it is the machine behind `run` asked a different question and contrasts with `peek`, making its purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context on when to use the tool: it works on any language or file type, including those without extensions or unknown to the system, and it contrasts with `peek` for full content. It implies the tool is for structural summaries rather than full reads, but it does not explicitly state 'when not to use' or name the exact alternative for each scenario. However, the contrast with `run` and `peek` gives strong guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• runA
  4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses significant behavioral traits beyond the annotations: it explains the output filtering mechanism (only relevant lines shown, gaps quantified), the persistence of full output on disk, the re-sending of results with the transcript, and the integration with `peek` for retrieving arbitrary ranges. It also clarifies the `keep` parameter's behavior. This exceeds what annotations (readOnlyHint=false, destructiveHint=true) already convey and does not contradict them.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded: it states the core purpose first, then explains the mechanism, then gives usage guidance. Every sentence adds value—there is no filler or repetition. Despite its length, it remains dense and useful, prioritizing the most critical information for an agent deciding whether and how to call the tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with six parameters and no schema descriptions, the description is incomplete. It covers the output-handling behavior thoroughly but leaves the semantics of `cwd`, `timeout`, `background`, and `budget` unspecified. An agent cannot fully understand the tool's capabilities or constraints without additional inference. The presence of an output schema helps with return values, but parameter completeness is still lacking.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate for all six parameters. It provides detailed semantics for `keep` ('every line containing it comes back whatever else was chosen') and mentions `peek` with a returned handle, but it says nothing about `cwd`, `timeout`, `background`, or `budget`. This leaves a large gap in parameter understanding, especially for a tool with destructive potential.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb and resource: 'Run a shell command and return only the lines that mattered.' It distinguishes itself from a generic shell tool by emphasizing filtered output, and it names the sibling 'peek' as a companion. However, it does not directly differentiate itself from the listed siblings (follow, outline, digest, tool, digest_many), relying instead on the general notion of a 'plain shell tool'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit guidance: 'Prefer this over a plain shell tool whenever the output may be long or noisy.' It also provides a concrete usage pattern: 'When you already know what you are looking for -- a symbol, a test name, an error code -- pass it as `keep`.' This tells the agent when to use the tool and how to tailor its invocation, leaving little to inference.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• followA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the annotations (destructiveHint: true), the description discloses specifics: `stop` 'ends it, and everything it started,' and a left-alone background command keeps running (openWorldHint). It also reveals the subtle consequence that an empty answer stays in the conversation for the rest of it, which is not captured by any annotation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The first sentence front-loads the core purpose, and the rest is dense, informative prose covering modes, parameters, and behavioral caveats rather than padding. It is longer than average, but the tool has four interacting modes and subtle edge cases, so the length is justified.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description explains the return format (incremental output since last ask, preserved line numbers, gap summaries for progress-only stretches, last output on stop) and covers the tricky consequences (empty results persisting in the conversation, background commands continuing to run). With an output schema present and all four parameters addressed, the description is complete enough for correct invocation in the described scenarios.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has zero description coverage, so the description carries the full burden and does address all four parameters in prose: `handle` (from a `run` call with `background`), `stop` (ends the command and what it started), `everything` (answers for every run still going), and `wait` (holds until something is said). However, some details remain implicit: the units of the `wait` number and the behavior of a null `handle` with no other flags are not specified.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The first sentence states a specific action (return what a background command has printed since last asked) on a specific resource (a background command from `run`), which clearly distinguishes it from siblings like `peek` (which fetches a specific line) and `run` (which starts a background command). The title and description align on the core purpose of following an ongoing background run.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly ties the tool to a `run` call made with `background` and contrasts it with `peek`, explaining the conditions for each (lines keep their numbers, so `peek` returns the same line). It gives concrete when-to-use guidance for `everything` ('when you started three things and want to know where they are') and for `wait` (avoiding empty results that persist in the conversation), plus the imperative to `stop` a finished command.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.