Server Quality Checklist

Tool Scores

• web_fetchC
  2.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided. Description only mentions 'extracted text' but lacks details on HTTP methods, redirects, timeouts, or truncation behavior. Default max_chars parameter not explained.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence is concise but omits important details. Minimal structure adequate for a very simple tool, but missing parameter context.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Lacks context for alternatives (browser_fetch) and does not leverage output schema to describe return value. Insufficient for agent to use accurately.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% and description adds no information about parameters 'url' or 'max_chars'. Agent has no guidance on input semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states verb 'Fetch' and resource 'URL' with outcome 'return its extracted text'. No differentiation from sibling 'browser_fetch'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool vs alternatives like 'browser_fetch'. Missing context about prerequisites or use cases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• github_searchC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must bear the full burden of behavioral disclosure. It only states 'read-only', which is a positive trait, but omits important details such as authentication requirements, rate limits, output format, or pagination behavior. This is insufficient for a search tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single short sentence, which is concise. However, conciseness is only beneficial when the core information is present. Here, it omits critical details, making it under-specified rather than efficiently packed.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 2 parameters (one required), an output schema, and no annotations, the description is incomplete. It fails to explain parameters, return values, or usage context. While the sibling list provides some domain context, the description itself leaves significant gaps for the agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, meaning the description must explain parameters. It does not mention 'query' or 'limit' at all, nor their formats, defaults, or constraints. The schema provides basic structure, but the description adds no semantic value, leaving the agent to guess parameter usage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Search' and the resource 'GitHub issues and pull requests', and adds 'read-only' for behavioral context. It is specific enough to distinguish from sibling tools like memory_search or web_fetch, but could be more precise about the scope (e.g., whether it includes comments or only issues/PRs).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no explicit guidance on when to use this tool versus alternatives. It does not mention prerequisites, exclusions, or compare with siblings like memory_search or web_fetch. The agent must infer appropriateness from the name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• memory_searchC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must cover behavioral traits. It does not disclose case sensitivity, whether it searches all users or current session, or if it returns full entries or excerpts. Minimal transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with only 6 words, making it very concise. It is front-loaded with the key action. However, the extreme brevity sacrifices necessary detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations, missing parameter semantics, and no usage guidance, the description is incomplete for an agent to correctly select and invoke this tool. The presence of an output schema does not compensate for these gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has two parameters (query, limit) with 0% schema description coverage, but the description adds no explanation of their meaning, syntax, or valid values. The description relies entirely on the schema, which lacks human-readable details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool performs keyword search over ACR's local memory store, distinguishing it from sibling tools like skill_search or web_fetch. The verb 'search' and resource are specific, though 'ACR' is not expanded.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus alternatives. The description implies it is for local memory search, which is a basic context, but lacks exclusion criteria or links to sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• skill_searchB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description bears full responsibility for behavioral disclosure. It only states 'keyword search,' omitting details like exact vs. fuzzy matching, result format, pagination behavior, or error handling. This is insufficient for a search tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, highly concise with no filler. However, it could be structurally improved by front-loading more context or breaking into multiple points for clarity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has an output schema, reducing the need to explain return values. However, the description lacks usage context such as when to use this vs. memory_search or web_fetch, making it adequate but not complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, meaning the description must compensate. It adds no insight beyond the parameter names: 'query' and 'limit.' No format, restrictions, or default behavior is described, so it fails to enrich the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Keyword search over ACR's local skill registry,' identifying the verb (search), resource (skill registry), and scope (ACR's local). This distinguishes it from sibling tools like web_fetch and memory_search, which have different domains.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. There is no mention of use cases, exclusions, or prerequisites, leaving the agent to infer context from the name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_taskB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided; the description mentions the mock provider behavior but fails to disclose potential side effects, authentication needs, rate limits, or what happens to data during execution.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences with no unnecessary words. However, the structure could be improved by front-loading the purpose more explicitly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations, incomplete parameter descriptions, and no explanation of return values despite an output schema, the description leaves significant gaps in understanding the tool's behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%; the description only alludes to min_quality_tier without defining it or the 'objective' parameter. It does not add meaningful meaning beyond the schema's property names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the verb ('Run') and the resource ('objective through ACR's task engine'). The tool's purpose is distinct from sibling tools (search, fetch), making selection unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides context on provider selection based on min_quality_tier or default configuration, but lacks explicit guidance on when to use this tool versus alternatives or when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• browser_fetchB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It mentions using a 'real headless browser' but does not disclose behavioral traits such as execution time, resource usage, error handling, or side effects. This leaves significant gaps in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that is concise and front-loaded with the main action. It is efficient but could include a bit more detail without being verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of a headless browser tool and the existence of an output schema, the description covers the key differentiator from siblings. However, it omits information about error conditions, limitations, and parameter details, leaving it insufficient for complete context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It does not explain the purpose or constraints of the 'max_chars' parameter, and only indirectly references the 'url' parameter by mentioning it. No additional semantic value is added beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool renders a URL in a headless browser and returns visible text, including JS-rendered content. It distinguishes itself from the sibling 'web_fetch' by noting that web_fetch cannot see JS-rendered content, so the purpose is specific and well-differentiated.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for pages with JS-rendered content that web_fetch cannot access, providing clear context. However, it does not explicitly state when not to use it or mention alternatives beyond web_fetch, so slightly lacking in complete guidelines.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.