Server Quality Checklist

Tool Scores

• doc_searchB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses some behavioral traits like case-insensitive search, multi-keyword AND logic, and supported file extensions, but misses critical details such as error handling (e.g., invalid paths), performance limits, output format specifics, or whether it's read-only/destructive. This leaves gaps for an agent to use it correctly.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, starting with the core purpose. It uses two sentences efficiently, though the second sentence is slightly dense with file extension details. Overall, it avoids unnecessary repetition and wastes little space.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description is moderately complete for a search tool but has gaps. It covers what the tool does and some behavioral aspects, but lacks details on output format, error conditions, and performance constraints, which are important for an agent to invoke it effectively without structured output guidance.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description adds minimal value beyond the schema, mentioning 'case-insensitive' (covered by case_sensitive default) and 'searchable extensions' (implied by file_extension_filter), but does not provide additional syntax or format details. Baseline 3 is appropriate as the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('searches all documents') and resources ('local docs/ folder'), and distinguishes it from siblings by specifying it searches documents rather than executing code, reading repos, or running terminal commands. It provides concrete details about supported file types and search behavior.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for searching documents in a folder, but does not explicitly state when to use this tool versus alternatives like sibling tools (e.g., github_repo_reader for remote repos). It mentions the default directory ('docs/') but lacks explicit guidance on prerequisites or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• github_repo_readerA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes key behaviors: recursive traversal, automatic ignoring of specific directories and file types, handling of binary and large files, and the return format (directory tree and file contents). This gives the agent clear expectations without contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose in the first sentence, followed by essential behavioral details in a compact form. Every sentence adds value (e.g., filtering rules and return format), with no redundant or vague language, making it highly efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (recursive file reading with filters), no annotations, and no output schema, the description does a good job covering behavior and output. It explains what is returned (directory tree and file contents) and key constraints, though it could benefit from mentioning error handling or performance implications for large repositories.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters thoroughly. The description does not add any additional meaning or context beyond what the schema provides for 'repo_path' and 'max_files', such as usage examples or constraints not in the schema. Baseline 3 is appropriate as the schema handles parameter documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('reads', 'returns') and resources ('all source files in a local repository directory'). It distinguishes itself from potential siblings by specifying recursive reading with automatic filtering of common directories and file types, which is not implied by the tool name alone.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context through details like 'local repository directory' and filtering of '.git', 'node_modules', etc., suggesting it's for analyzing codebases. However, it does not explicitly state when to use this tool versus alternatives like 'doc_search' or 'code_executor', nor does it provide exclusions or prerequisites for usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• code_executorA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and effectively discloses key behavioral traits: it specifies hard limits (15-second timeout, 64 KB output cap), isolation (runs in isolated temp file), and statelessness (no persistent state between calls). It does not cover aspects like error handling or security implications, but provides substantial operational context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with core functionality and efficiently structured in three sentences: the first states purpose and constraints, the second details isolation, and the third provides usage context. Every sentence adds value without redundancy, making it appropriately sized and zero-waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (code execution with constraints), no annotations, and no output schema, the description is largely complete: it covers purpose, behavioral limits, and usage context. However, it lacks details on return values or error responses, which would be helpful for an agent invoking the tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters thoroughly. The description adds minimal value beyond the schema, mentioning 'self-contained' for the code parameter but not elaborating on syntax or constraints. Baseline 3 is appropriate as the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool executes code snippets in specific languages (Python/Node.js) in a sandboxed environment, distinguishing it from sibling tools like doc_search or terminal_commander by focusing on code execution rather than document retrieval or system commands. It specifies the action ('executes'), resource ('code snippet'), and context ('sandboxed child process').

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool ('ideal for quick calculations, data transformations, and API-free logic'), but does not explicitly state when not to use it or name alternatives among siblings. It implies usage for small, stateless tasks but lacks explicit exclusions or comparisons to other tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• terminal_commanderA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and does an excellent job disclosing key behavioral traits: safety mechanisms (strict allowlist, blocked dangerous patterns), platform specifics (Windows CMD/PowerShell), and examples of allowed commands. It doesn't cover error handling or output format, but provides substantial operational context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded with the core purpose and safety constraints, followed by specific examples. Every sentence adds value, though the list of allowed commands could be slightly more concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no annotations and no output schema, the description provides excellent context about safety constraints, allowed commands, and usage examples. It doesn't describe the return format or error behavior, but covers most essential operational aspects given the complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already fully documents all three parameters. The description adds no additional parameter information beyond what's in the schema, meeting the baseline expectation when schema coverage is complete.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('executes') and resources ('Windows CMD or PowerShell command on the local machine'), and distinguishes it from siblings by focusing on safe command execution rather than code execution, document search, or GitHub operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly defines when to use this tool ('to run git status, dir, npm install, tsc --noEmit, etc.') and provides clear alternatives by naming sibling tools (code_executor, doc_search, github_repo_reader), though it doesn't explicitly state when not to use it beyond the allowlist constraints.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.