Server Quality Checklist

Tool Scores

• list_categoriesA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  There are no annotations, so the description carries the full burden of disclosing behavior. It implies a read-only operation by saying 'overview' but does not explicitly state that no modifications occur, nor does it mention any permissions, side effects, or return format. This is a minimal disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that conveys the essential purpose with no filler. Every word adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list tool with no output schema, the description adequately covers the main purpose and output scope. It could mention the format of the counts or categories, but for the low complexity, it is sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and the baseline for this case is 4. The description does not need to add parameter-specific information, as there is nothing to clarify.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool provides an overview of every yala toolkit category with tool counts. It uses a specific resource ('yala toolkit category') and identifies the output scope ('with tool counts'), making it distinct from siblings like find_tools or tool_info.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. It does not mention scenarios where find_tools or tool_info would be more appropriate, nor does it specify any prerequisites or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_toolsA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavior. It states it returns ranked matches with descriptions, which gives the return format. However, it does not mention whether the operation is read-only, any authentication requirements, or potential errors. For a search tool, this is adequate but not highly transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise (two sentences) and front-loaded with the core action and examples. 'Start here' is a useful directive. No unnecessary words or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (3 params, no output schema), the description covers what it does, gives concrete examples, and specifies the return type (ranked matches with descriptions). The schema handles parameter details, so the description is complete for an agent to correctly invoke it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with clear descriptions for each parameter (query, limit, category). The tool description adds no extra parameter information beyond the schema, but the schema already provides sufficient meaning. Baseline of 3 applies because the description does not need to compensate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: to search the yala toolkit's 500+ CLI utilities by task. It uses a specific verb ('search') and resource ('yala toolkit's 500+ standalone CLI utilities'), and includes examples. This distinguishes it from siblings like tool_info (specific tool details), run_tool (execution), and list_categories (category listing).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'Start here' clearly positions this as the entry point for finding tools, implying it should be used before other tools when you need to locate a utility. However, it does not explicitly state when not to use it or mention alternative tools, though the context provides enough guidance for an agent.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_toolA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It transparently discloses the return format (exit code, stdout, stderr, timing), the no-shell argument behavior, and the dry-run support for mutating tools. This is substantial for an execution tool, though it could mention error handling or permissions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, front-loaded with the primary purpose, and every sentence contributes useful information. No redundant phrases, and the dry-run tip is valuable without bloating the text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema and no annotations, the description provides necessary return-value information (exit code, stdout, stderr, timing) and context (toolkit root, no shell). It covers the main execution aspects, though it could arguably mention what happens on timeout or failure, which is partially covered by timeout_s in the schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 75% (3/4 params described). The description adds meaning beyond the schema by clarifying that args are an argv list (no shell) and that --dry-run is a preferred flag for mutating tools, which contextualizes the args parameter. This adds value over the schema alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb (execute) and resource (a yala tool), and specifies what it returns (exit code, stdout, stderr, timing). This distinguishes it from sibling tools like find_tools and tool_info, which are for discovery rather than execution.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides a clear usage guideline: prefer --dry-run for tools with mutates=true. While it does not explicitly name alternatives, it is the only execution tool among siblings, and the dry-run guidance is directly actionable, implying when to use this tool for safe testing first.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• tool_infoA
  4.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations are absent, so the description carries the burden. It mentions 'live --help' and 'whether it mutates state', but does not explicitly state whether tool_info itself has side effects or is read-only. The instruction 'Use before run_tool' hints it's safe, but not explicitly confirmed, leaving some ambiguity.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences that list what the tool provides and when to use it. No unnecessary words or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description explains the return content (description, examples, help, mutation status) and the intended usage context. It is complete for a retrieval tool without needing error details.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Only one parameter 'name' with a clear description and example ('openapi_diff.py'). The schema covers 100% and the example enhances understanding, so it's above baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: to provide full detail about a single tool, including description, example commands, live --help, and mutation status. This distinguishes it from siblings like find_tools (discovery), run_tool (execution), and list_categories (categorization).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says 'Use before run_tool', giving a clear directive on when to call this tool. It also implies it's for retrieving details about a specific tool, contrasting with find_tools for searching and list_categories for listing categories.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.