Server Quality Checklist

Tool Scores

• search_githubC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It mentions what fields are returned (repo name, description, stars, language, topics) but doesn't cover important aspects like rate limits, authentication requirements, pagination behavior, or error conditions for a search API tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately brief (two sentences) and front-loaded with the core purpose. Every sentence adds value: the first states what the tool does, the second describes the return format.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a search tool with 2 parameters, no annotations, and no output schema, the description is insufficient. It doesn't cover authentication needs, rate limits, error handling, or how results are sorted/filtered. The return format is mentioned but without schema details.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 50% (only 'query' has a description). The description doesn't add any parameter-specific information beyond what's in the schema. It doesn't explain search query syntax, result ordering, or what 'num_results' default of 10 means in practice.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Search GitHub repositories') and the resource ('GitHub repositories'), making the purpose immediately understandable. However, it doesn't differentiate from sibling tools like 'search_google' or 'search_web' beyond mentioning GitHub specifically.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus the sibling search tools (compare_engines, search_academic, search_google, search_web). It mentions GitHub but doesn't explain why one would choose GitHub search over other search options.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_academicB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It mentions the sources (arXiv, Wikipedia) but doesn't describe important behaviors like rate limits, authentication needs, result format, pagination, or whether this is a read-only operation. The description is insufficient for a tool with no annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise with just two sentences that are front-loaded and waste-free. The first sentence states the core purpose, and the second provides usage context. Every word earns its place with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations, no output schema, and incomplete parameter documentation (50% schema coverage), the description is insufficiently complete. It doesn't explain what the tool returns, how results are structured, or important behavioral constraints. For a search tool with multiple sibling alternatives, more context is needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 50% (only 'query' has a description). The description adds no specific parameter semantics beyond what the schema provides. It doesn't explain what constitutes a good query format, what 'num_results' controls, or any constraints. With moderate schema coverage, the baseline 3 is appropriate as the description doesn't compensate for the coverage gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose as 'Search academic sources (arXiv, Wikipedia)' with specific resources named. It distinguishes from siblings by focusing on academic sources rather than general web, GitHub, or engine comparison. However, it doesn't explicitly contrast with each sibling tool by name.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides implied usage guidance with 'Best for research questions, paper discovery, and factual lookups,' suggesting appropriate contexts. However, it doesn't explicitly state when NOT to use this tool or name specific alternatives among the sibling tools (compare_engines, search_github, search_google, search_web).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_webB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It mentions 'aggregated, deduplicated results' and 'unified schema,' which adds some behavioral context, but fails to disclose critical traits such as rate limits, authentication needs, error handling, or what 'enabled providers' entails. This is a significant gap for a web search tool with no annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core functionality and followed by a usage hint. Every word earns its place, with no redundancy or waste, making it highly efficient and easy to scan.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of a web search tool with 5 parameters, no annotations, and no output schema, the description is incomplete. It lacks details on result format, error cases, provider specifics, and behavioral constraints, making it inadequate for safe and effective use by an AI agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 40%, with only the 'query' parameter having a description. The description adds no specific parameter semantics beyond what the schema provides, such as explaining 'providers' options or 'language'/'country' effects. It compensates minimally, so the baseline 3 is appropriate given the low coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Search') and resource ('the web'), specifying it uses 'all enabled providers' and returns 'aggregated, deduplicated results with a unified schema.' It distinguishes from siblings by mentioning 'general queries,' but could be more explicit about how it differs from specific providers like search_google or search_academic.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for 'general queries,' which suggests when to use this tool, but does not explicitly state when not to use it or name alternatives. It lacks clear guidance on choosing between this and sibling tools like search_google or search_academic, leaving usage context somewhat vague.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compare_enginesA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions the tool runs queries and returns grouped results, but does not cover critical aspects like whether this is a read-only operation, potential rate limits, authentication needs, error handling, or what happens when providers fail. For a tool that interacts with multiple external services, this is a significant gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that efficiently conveys the tool's purpose and outcome without unnecessary words. It is front-loaded and every part earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of querying multiple providers, no annotations, and no output schema, the description is incomplete. It lacks details on behavioral traits, error scenarios, output format, and how results are structured for comparison. This is inadequate for a tool with external dependencies and multiple parameters.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 67% (2 out of 3 parameters have descriptions). The description adds value by explaining the purpose of comparing providers and implying the 'providers' parameter's role, but does not detail the 'query' or 'num_results' beyond what the schema provides. With moderate coverage, it compensates somewhat but not fully.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Run the same query against multiple providers') and the outcome ('return results grouped by provider for side-by-side comparison'), distinguishing it from sibling tools that search specific platforms. It uses precise verbs and identifies the resource being compared.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for comparative analysis across providers, but does not explicitly state when to use this tool versus the sibling search tools (search_academic, search_github, etc.). It lacks guidance on alternatives or exclusions, leaving the context somewhat implicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_googleA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses the return format ('structured organic results, answer boxes, and related searches') which is valuable behavioral information. However, it doesn't mention rate limits, authentication needs, error conditions, or pagination behavior that would be helpful for a search tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is perfectly concise with two sentences that each earn their place. The first sentence establishes the core functionality and constraints, while the second specifies the return format. No wasted words, front-loaded with essential information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a search tool with 3 parameters, no annotations, and no output schema, the description provides adequate but incomplete context. It covers the basic purpose and return format, but lacks details about error handling, rate limits, provider differences, or what happens when no results are found. The absence of output schema means the description should ideally explain more about the return structure.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 67% schema description coverage, the description adds meaningful context beyond the schema. While the schema documents parameters, the description clarifies that providers are 'serpbase or serper' (matching the enum) and that results include 'organic results, answer boxes, and related searches' - giving semantic meaning to the search operation that the schema alone doesn't provide.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Search Google'), identifies the resource ('via a configured third-party provider'), and distinguishes from siblings by specifying it's for Google searches only, unlike 'search_academic' or 'search_github'. It provides verb+resource+scope differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context by specifying it's for Google searches via particular providers, which helps differentiate from sibling tools like 'search_academic'. However, it doesn't explicitly state when to use this versus alternatives or provide exclusion criteria, leaving some ambiguity about provider selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.