Server Quality Checklist

Tool Scores

• edit_fileA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses UTF-8 handling, newline convention, and match constraints (exact, unique unless replace_all). However, it omits permissions, error handling, and whether the file is created if missing.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no redundancy. Every word is informative. Front-loaded with the core operation and key constraints.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given an output schema exists, the description covers operation, constraints, and a usage hint. It mentions UTF-8 and newline convention. Could add error handling or file existence requirement, but overall adequate for a well-defined edit tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has 0% description coverage, so the description compensates partially. It explains old_string (non-empty, exact match) and replace_all (changes behavior). Path and new_string are implied but not detailed. Adds moderate value beyond names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it performs search-and-replace on an existing UTF-8 file, which distinguishes it from siblings like read_file, write_file, and run_command. It specifies the action and resource concisely.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance: 'Read the file first and copy the exact span.' It also clarifies conditions (old_string must match exactly once unless replace_all is true). However, it does not explicitly contrast with write_file for full rewrites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• write_fileA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses mode (overwrite/append) and that parent directories are created, but lacks details on error handling, file locking, atomicity, or encoding guarantees. Adequate but not comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, no excess. Front-loaded with main action, then concise usage warning. Every word adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 3 params and output schema, the description covers key behavioral aspects (mode, dir creation, usage caution). Minor missing details like error conditions or concurrency behavior, but mostly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so description must compensate. It adds meaning for mode (default and options), path (under workspace), and content (UTF-8). However, it lacks details on path constraints, content size limits, or format of mode enum. Partial compensation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the verb (write/append), resource (UTF-8 text under workspace), and mode distinction. Differentiates from siblings like read_file and edit_file through the write/append and create parent dirs functionality.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly warns not to use without reading first unless creating a new file or holding exact contents, providing clear when-to-use guidance. Could be improved by explicitly contrasting with edit_file, but still effective.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_filesA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries burden. It describes listing behavior and optional parameters but doesn't disclose details like performance, permissions, or output format. Adequate but minimal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with action, no wasted words. Every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list tool with output schema, description is mostly complete. It mentions optional parameters and usage context. Could note sorting or limits, but overall sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so description must compensate. It lists parameter names (recursive, glob_pattern, max_depth, include_dotfiles) but doesn't explain their semantics beyond names. Adds some value but insufficient detail.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states verb 'List' and resource 'files and directories under a workspace path'. It distinguishes from sibling tools (read_file, write_file, etc.) by positioning itself as a discovery tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states 'Use to discover layout before reading', providing clear context and guidance on when to use this tool relative to reading files.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_commandA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It mentions 'no shell' which is important for security, and cwd constraints. However, it lacks disclosure of potential side effects, output handling, or timeout behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no wasted words, front-loading the core action and usage guidance.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 3 parameters and no annotations, the description covers argv and cwd adequately. The timeout parameter is undocumented, but the presence of an output schema may compensate for return behavior. Overall, sufficient for command execution.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so description must compensate. It adds meaning for argv (list only) and cwd (relative or absolute under root), but provides no information about the timeout parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool runs a subprocess with an argv list and no shell, and specifies cwd constraints. It distinguishes from sibling tools (read_file, list_files) by explicitly advising to prefer those for inspection.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Use only when necessary; prefer read/list for inspection', providing clear when-to-use guidance and directing to alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• read_fileA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It specifies that the file is UTF-8 text, and mentions optional line range (1-based inclusive) and max_bytes override. It does not detail error handling or behavior for non-existent paths, but the core behavior is well communicated.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long, front-loading the purpose and file type, followed by usage guidelines and parameter details. Every sentence earns its place with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool is a simple file reader with an output schema (not shown), the description covers the main usage, line range, and byte limit. It is mostly complete but could mention error cases or binary file handling for full completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema coverage is 0%, so the description must add meaning. It does so by explaining all parameters: path (required), start_line/end_line (1-based inclusive), and max_bytes override. This adds significant value beyond the schema's type information.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Read a UTF-8 text file under the workspace' with a specific verb and resource. It distinguishes itself from siblings like write_file, edit_file, run_command, and list_files.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Use before editing when content is unknown', providing clear context for when to use the tool. It does not include explicit exclusions or alternatives, but the guidance is sufficient.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.