Server Quality Checklist

Tool Scores

• scrape_urlC
  2.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must disclose behaviors itself. It mentions the output format (markdown) and method options, but lacks details on error handling, rate limits, or side effects of scraping. The short description leaves significant behavioral aspects unmentioned.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise, front-loading the core function. However, it could benefit from a bit more detail without becoming verbose, so not a 5.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations, 0% schema coverage, and a tool with multiple method-specific behaviors, the description is incomplete. It does not clarify return structure beyond 'markdown content' or provide enough to correctly select between methods.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description names the 'method' parameter and its three options, but does not explain the differences between them or the expected format of 'url.' Since schema coverage is 0%, the description only partially compensates.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states 'Scrape a single URL and return markdown content,' which clearly identifies the tool's action and output. It differentiates from siblings like 'search_all' by specifying 'single URL,' though it doesn't explicitly contrast with alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus the siblings (research, search_all, etc.). The description only states what it does, not the context or comparison.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_allC
  2.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It does disclose that results are deduped, which is useful, but it does not mention any other behavioral traits such as whether the operation is read-only, whether it requires authentication, how errors are handled, or what 'every source' concretely means. This is a minimal level of transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear, and front-loaded sentence with no wasted words. It conveys the core action and result type efficiently. While it is short, conciseness is about structure and economy, which this achieves perfectly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Although the tool has an output schema to describe return values, the description is incomplete for a tool with 3 parameters and no annotations. It lacks usage guidance, parameter semantics, and a clear definition of 'every source'. It does not explain the max_results parameter or how sources can be restricted, making it inadequate for an agent to select and invoke this tool confidently.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, and the description does not compensate by explaining any of the three parameters (query, sources, max_results). It does not even implicitly clarify the meaning of 'sources' beyond the phrase 'every source' or mention the max_results limit. The agent must rely solely on parameter names and defaults, which is insufficient for correct invocation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly indicates the tool fans out a query to every source and returns deduped SearchResult dicts, which distinguishes it from the sibling tools (research, scrape_url, rerank_docs, summarize_docs). The verb 'fan out' is specific, and the resource ('every source') is named. However, it does not explicitly say 'search', relying on the word 'query' and the return type to convey the core purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is given about when to use this tool vs. alternatives. It does not mention any sibling tools, nor does it state explicit conditions such as 'use when you need results from all sources'. The phrase 'every source' implies comprehensive search, but there is no clear 'when' or 'when not' guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• summarize_docsB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  There are no annotations, so the description must carry the full transparency burden. It fails to disclose behavioral traits such as whether this is a read-only operation, whether it calls an LLM API (cost/latency implications), or how the model/style parameters affect the output. The mention of model options hints at external AI usage but doesn't explicitly state behavior or side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise, using two sentences and a compact param list. Every element adds value—first sentence states purpose, the list documents available options. No filler or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations and low schema coverage, the description is minimally viable but not complete. It defines the tool's action but lacks context on when to deploy it relative to sibling tools, how the query parameter is used, or what the output format looks like despite the output schema being present. It covers the basics but leaves gaps for an agent to infer.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description must explain parameters, but it only lists possible values for style and model, omitting the meaning or role of the required 'query' parameter. It partially compensates by specifying style options, but the critical 'query' param is left undefined, leaving a significant semantic gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Synthesize a final answer from a list of document strings' uses a specific verb (synthesize) and resource (document strings), clearly distinguishing it from sibling tools like research or scrape_url. The purpose is unambiguous and immediately conveys the core function.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Usage is implied by the description: you should use this when you have document strings and need a synthesized answer. However, there is no explicit when-to-use, exclusions, or alternatives mentioned relative to sibling tools such as research or rerank_docs, so the guidance remains implicit rather than instructive.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• rerank_docsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full behavioral disclosure burden. It does disclose the return format (sorted descending) and the accepted model identifiers, but it omits potential side effects, rate limits, or the fact that using remote models like cohere/jina may incur network calls and costs.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise and well-structured. It leads with the core action, then lists model options and the return format. Every sentence contributes value with no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (3 parameters, simple return array), the description is largely complete. It covers the purpose, parameter semantics implicitly, and return shape. It lacks usage guidance and error handling, but these are not critical for a straightforward rerank operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, but the description enriches the parameters by explaining 'docs' as document strings, 'query' as the relevance basis, and explicitly listing the allowed model values. It adds semantics beyond the raw schema, though it could further elaborate on query expectations.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: reranking document strings by relevance to a query. The verb 'rerank' is specific and distinct from sibling tools like search_all or summarize_docs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Usage context is implied: you must already have a list of documents and a query to rerank. However, the description does not explicitly state when to use this tool over alternatives, nor does it provide exclusions or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• researchA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It transparently lists the sequence of operations (search -> scrape -> rerank -> summarize), which is the core behavior. It does not mention rate limits, auth, or failure handling, but for a read-only research pipeline, the disclosed process is sufficiently transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, compact sentence that uses an arrow pipeline to efficiently convey the tool's flow. Every word is purposeful, with no fluff or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that this is a composite tool orchestrating four sibling sub-tools, the description effectively communicates the end-to-end nature. The output schema and fully documented parameters cover return values and inputs, so the description is complete enough for the agent to understand the tool's role, though a bit more context about typical use cases could be added.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with each parameter fully described in the input schema. The tool description itself adds no parameter-specific context beyond the pipeline, so it does not augment the schema's already comprehensive parameter definitions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states the tool performs an end-to-end pipeline: search all sources, scrape, rerank, and summarize. This clearly defines the tool's purpose and differentiates it from sibling tools like search_all or scrape_url, which cover only individual steps.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The 'end-to-end' phrasing implies that this tool is used when the user wants the full research workflow in one call. However, it does not explicitly mention when to prefer this over using the sibling tools individually, nor does it state any exclusions or alternative conditions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.