Server Quality Checklist

Tool Scores

• doctorC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description must carry the full burden. It suggests a read-only operation ('diagnose') but does not confirm idempotency, side effects, or authorization needs. Important behavioral traits are missing.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence. It is front-loaded, but the content is too minimal to earn a 5. There is no fluff, making it appropriately sized for a simple tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations, output schema, and parameter details, the description is the sole source of context. It does not explain what the tool returns, how to interpret results, or any operational constraints. A healthy provider check would benefit from at least indicating output format or typical usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are no parameters, so the schema coverage is effectively 100%. According to guidelines, baseline is 4 for zero params. The description does not need to add param details, and it correctly omits them. No contradiction.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a verb 'diagnose' and references 'provider availability and health,' but the meaning of 'diagnose' is vague. It is unclear whether this tool checks status, runs tests, or returns reports. It distinguishes from sibling tools like 'search' and 'extract' only implicitly, but not explicitly enough.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. The description does not mention prerequisites, context, or exclusions. The agent is left to infer usage from the vague verb 'diagnose'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• vertical_batchC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses parallel execution and a limit of 5, but no annotations exist. Missing information on safety (read-only vs destructive), authentication, error behavior, or result format. The description carries the full burden and is insufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, no waste, and front-loads the key constraint. Could be slightly expanded to include return behavior without losing conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has one complex parameter, no output schema, and no annotations. The description does not explain return values, error conditions, or what 'vertical queries' are. It is adequate for a simple tool but incomplete for practical use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with a clear description of each item's structure. The description adds 'in parallel' but that is behavioral, not about parameter semantics. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states it runs 'vertical queries in parallel' up to 5, clearly indicating the action and resource. However, it does not use a specific verb like 'execute' or 'run', and assumes familiarity with the term 'vertical queries'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus siblings like vertical_search, search, or extract. The phrase 'via AnySearch' provides context about the backend but not about selection criteria.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• extractA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden and effectively discloses the fallback order of extraction methods, which is useful for understanding reliability and behavior. However, it does not mention side effects or error handling.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that efficiently conveys the main purpose and the fallback chain. Every part earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the basic function but lacks details about return value format, handling of failures, or edge cases. Without an output schema, more context about what 'clean markdown' entails would improve completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Only 50% of schema parameters have descriptions (url has one, timeout does not). The tool description does not compensate by explaining timeout or adding meaning beyond the schema. It repeats that extraction is from a URL but adds no parameter-specific details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool extracts clean markdown from a URL, with a specific verb and resource. It distinguishes from sibling tools like search and doctor by focusing on content extraction.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for extracting markdown from URLs but does not explicitly say when or when not to use this tool versus alternatives. No comparison to sibling tools is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• searchA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It explains parallel provider calls, URL deduplication, RRF reranking, and snippet cleaning. However, does not mention rate limits or auth needs, though irrelevant for a read-only search.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, no redundancy, front-loaded with key purpose. Every word adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  No output schema exists, and description does not explain return format (e.g., fields in results, pagination). It mentions 'snippets cleaned' but not what is returned. Could be more complete for a search tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, baseline 3. The description adds context about providers being tried in parallel and RRF fusion, which isn't in the schema. Thus adds meaningful behavioral context beyond parameter definitions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool performs multi-source web search with RRF fusion, specifying providers and processing details. It distinguishes from vertical_search siblings by indicating it's a general web search.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use vs siblings like vertical_search or doctor. Usage is implied from the description of multi-source web search, but no alternatives or when-not-to-use are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• vertical_searchA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Mentions returns structured payload, but lacks details on read-only nature, authentication, rate limits, or any side effects. With no annotations, more behavioral context would be helpful.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with front-loaded purpose and examples, no redundancy, every part earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers purpose and domain examples well, but given the complexity (5 params, nested objects, no output schema), lacks details on return format, pagination, or error handling.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Adds meaning beyond schema by clarifying domain enum values (e.g., finance for tickers) and notes sub_domain_params must have required keys. Schema coverage is high, but description enhances usability.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states it's a vertical-domain structured search tool with specific examples (finance, academic, security) and distinguishes from siblings like 'search' and 'vertical_batch'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Tells when to use (specific domains) and implies alternatives via sibling list, but doesn't explicitly state when not to use or mention alternatives within the description.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• vertical_sub_domainsA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description must cover behavior. It implies a read operation ('Discover'), but does not explicitly state it's non-destructive, mention return format, or disclose any side effects or authorization needs.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the action verb and resource. No unnecessary words; each sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description does not clarify return structure (e.g., list of subdomains). It also does not mention handling of multiple domains in 'domains' parameter or relation to other siblings like vertical_batch.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with parameter descriptions. The description adds value by explaining the purpose of the parameters (discovering sub-domains/params), giving context beyond what the schema provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool discovers sub-domains and required parameters for a vertical domain, and distinguishes it from the sibling 'vertical_search' by specifying when to call it.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says to call before vertical_search when sub_domain schema is unknown, providing good contextual guidance. However, it does not address when not to use the tool or mention other siblings like vertical_batch.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.