Server Quality Checklist

Tool Scores

• read_fileA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnly, idempotent, and non-destructive hints, so the safety profile is covered. The description adds useful context about UTF-8 encoding, line numbering, and offset/limit support, but doesn't disclose return format or error behavior, which would be valuable given no output schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the primary action and key details. Every sentence adds value: what it reads, the line numbering, the ToolAhead preference, and offset/limit support. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has three parameters and no output schema, so the description must explain not only parameters but also return behavior. It covers encoding, line numbers, and pagination, but omits what exactly is returned (e.g., an array of lines with numbers vs. a string), which is a notable gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already describes all three parameters with 100% coverage, including defaults and semantics for offset. The description adds minor context about offset/limit being useful for large files, but this is not essential given the schema's completeness.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool reads a UTF-8 text file from the workspace with 1-based line numbers, which is specific and distinguishes it from sibling tools like list_files and search. However, it references 'another file-read tool' without naming it, so it doesn't explicitly differentiate from all possible siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives a clear usage context: prefer this tool when available due to potential prefetched results from ToolAhead, and mentions offset/limit for large files. It also implies UTF-8 only, but doesn't explicitly state when not to use it (e.g., for binary files or other types of operations).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• write_fileA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds behavioral context beyond the annotations by specifying UTF-8 encoding, the fact that writes execute normally, and that they are 'never prefetched, cached, or replayed.' This is useful for an AI agent to understand execution semantics. It also discloses the overwrite behavior. There is no contradiction with the annotations; readOnlyHint=false aligns with the write nature.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no redundancy. The first sentence fronts the primary action and object, and the second adds essential execution context. Every word contributes to clarity, making it an excellent example of concise documentation.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple write tool with two parameters and no output schema, the description covers the core aspects: purpose, encoding, overwrite semantics, and execution guarantees. It does not mention return values or error handling, but these are minor for a write operation. The lack of an output schema means some gap exists, but overall it is sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema provides 100% coverage of both parameters with clear descriptions ('Complete UTF-8 file contents' and 'Absolute path or path relative to the workspace'). The description's phrase 'familiar file_path and content contract' adds no significant semantic detail beyond what the schema already conveys, hence the baseline score of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb and resource: 'Create or overwrite a UTF-8 workspace file'. This distinguishes it from siblings like read_file, list_files, search, and run, though edit_file is not explicitly differentiated. The mention of the 'file_path and content contract' adds scope clarity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage scenarios through 'Create or overwrite' and 'familiar file_path and content contract', suggesting it is for writing full file contents. However, it does not explicitly state when to prefer this tool over edit_file or other alternatives, nor does it provide exclusions or when-not-to-use guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• edit_fileA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds that edits 'execute normally and are never prefetched, cached, or replayed,' which is behavioral context beyond the annotations. The annotations already indicate non-read-only and non-idempotent, and the description is consistent, not contradictory.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences with the core action upfront. Every clause adds value—purpose, usage sequence, and execution semantics—with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given full schema coverage, annotations, and no output schema, the description covers the essential how and when to use, plus execution behavior. It doesn't detail error cases or return values, but those are not strictly required for a simple edit tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters. The description only names the parameters via 'familiar ... contract' without adding semantic details beyond the schema, hence a baseline score.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states 'Replace an exact string in a workspace text file,' which is a specific verb and resource. It also names the contract parameters and positions itself relative to read_file, distinguishing it from sibling tools like read_file and write_file.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It instructs to 'Use this ToolAhead edit after ToolAhead read_file,' providing clear usage context. However, it doesn't explicitly state when not to use it or contrast with alternatives like write_file, so it falls short of full exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• searchA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The annotations already indicate read-only, idempotent, non-destructive, and closed-world behavior. The description adds value by disclosing the output format (path:line:content matches) and the potential prefetched result behavior ('ToolAhead may return an exact prefetched result'). No contradictions with annotations are present.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is only two sentences long, front-loaded with the core purpose and output format, followed by a concise usage recommendation. Every sentence earns its place with no fluff or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (9 parameters, no output schema), the description provides essential context about what it does and the output shape. It does not repeat schema details but complements them with the prefetched result behavior and usage preference. Slightly more detail about the default output mode could push it to 5, but the schema covers that.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description does not add meaningful parameter semantics beyond what the schema already provides; it only reiterates that 'pattern' is a regular expression and mentions output as path:line:content, but the schema already covers these details. Thus, it neither enhances nor detracts from the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Search workspace file contents with a regular expression and return path:line:content matches.' It specifies the resource (workspace file contents), the action (search with a regular expression), and the output format. This effectively distinguishes it from sibling tools like read_file, list_files, edit_file, write_file, and run.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance to prefer this tool over other grep/search tools: 'Use this instead of another grep/search tool when available.' It also explains why (ToolAhead may return an exact prefetched result). However, it does not provide explicit exclusionary guidance for when to use read_file or list_files instead, so it falls short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_filesA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, openWorldHint, idempotentHint, and destructiveHint, so the main burden is on the description to add extra context. It adds a meaningful behavioral caveat: the tool 'may return an exact prefetched result,' which informs the agent about potential cached behavior beyond the standard annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences: the first states the core purpose, the second adds usage guidance and a behavioral caveat. Every word earns its place, with no filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity, the well-annotated schema, and the presence of safety annotations, the description is mostly complete. It covers purpose, usage, and the key prefetch behavior. A minor gap is not elaborating on what 'exact prefetched result' implies (e.g., freshness or staleness), but overall it is sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already describes 100% of parameters with examples and defaults, so the baseline is 3. The description adds value by clarifying that paths are 'project-relative', which is not fully explicit in the schema, and by establishing the workspace-scoped nature of the search.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Find') and resource ('workspace files') with a precise criterion ('project-relative paths match a glob pattern'). It clearly positions the tool as a file-listing tool and distinguishes it from the sibling read/edit/write/run tools, while also noting its advantage over another glob/file-listing tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly advises using this tool 'instead of another glob/file-listing tool when available' and gives a rationale ('ToolAhead may return an exact prefetched result'). It provides clear usage context but does not name the alternative sibling (e.g., search) or specify when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• runA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the idempotentHint annotation, the description discloses that ToolAhead may replay a prefetched result and that toolahead.toml commands get their services started and health-checked instead of being replayed. It does not contradict the annotations; a small gap remains around return output/error behavior, but the key behavioral traits are covered.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three short sentences, front-loaded with the primary purpose, followed by the replay nuance and an explicit fallback. No filler words; every clause contributes.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no output schema and a moderate annotation set, the description covers main use cases, replay behavior, and alternatives. It does not describe the return format or failure modes, but for a command runner the core behavior is sufficiently specified.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already covers both parameters, so baseline is 3. The description adds meaning by explaining that the command must be exact/deterministic and by expanding accepted commands to include those declared in toolahead.toml, which is not captured in the schema's command description. It doesn't fully specify how toolahead.toml commands are formatted, but enough is added.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Run an exact deterministic test, build, or lint command that is listed in .prefetch-replay.json.' It clearly distinguishes this from sibling file tools by limiting to replayable commands and directing other commands to the native shell.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives explicit when-to-use guidance: commands in .prefetch-replay.json or [commands] in toolahead.toml; it also states when not to use it: 'Use the agent's native shell for commands that are neither.' This is a clear exclusion rule.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.