Server Quality Checklist

Tool Scores

• list_sourcesA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations declare readOnlyHint=true, and the description adds capabilities context. However, it omits the dynamic health probe behavior triggered by check_health parameter, which is a behavioral trait beyond the static listing.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence of 12 words, front-loading the action. Every word adds value without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list tool with an output schema and one optional parameter, the description covers the main function. It could mention the optional health probing for completeness, but overall it is adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% (one parameter with full description). The tool description adds no additional meaning beyond the schema for the parameter, meeting the baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'List wired data sources and their capabilities' with a specific verb and resource. It distinguishes from siblings like search, resolve, fetch, and operate, which imply different actions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides the primary action but lacks explicit guidance on when to use this tool vs alternatives. There is no 'when not to use' or mention of sibling tools for different use cases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• resolveA
  4.3/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the readOnlyHint=true annotation, the description details behaviors of all optional parameters (e.g., 'fair is pure/local, no network call', 'trust only for DOI-bearing records', 'failures degrade quietly') and explains auto-attachment in provenance mode, offering comprehensive transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose and efficiently lists parameters with concise explanations. Though lengthy, it packs necessary detail without redundancy. Slightly more brevity could improve score.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (6 parameters, optional nested outputs) and the presence of an output schema, the description covers all intended behaviors and edge cases. It does not explicitly describe the base return structure, but output schema likely fills that gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, providing baseline 3. The description adds extra context for each parameter (e.g., 'use' explains verdict logic and matrix, 'cite' lists supported formats and fallbacks), elevating clarity beyond the schema alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Fetch the full DataResource for a known id', providing a specific verb and resource, and includes example IDs, which clearly defines the tool's purpose and distinguishes it from siblings like search (query-based) and fetch (likely simpler).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The intended usage (resolve by known ID) is implicit from the purpose, but the description does not explicitly contrast with siblings or state when not to use it. No exclusions or alternative tool mentions are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• fetchA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds substantial behavioral context beyond annotations: it describes checksum verification, sidecar writing, failure behavior on size/backends, and unverified sources. There is no contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long but well-organized: it front-loads the core purpose, then lists backends with verification status, then general behavior. Every sentence adds information, though it could be slightly more concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (many backends, multiple parameters, output schema exists), the description is comprehensive. It covers backend-specific behavior, failure modes, sidecar file, and checksum verification, with no obvious gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters. The description adds global context about 'max_bytes' and 'force' but does not significantly enhance individual parameter semantics beyond what the schema provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it downloads a resource's files to local disk and returns paths, not contents. It lists specific backends, distinguishing this tool from siblings like search or resolve.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit context about when to use this tool (e.g., downloading from various backends) and mentions failure conditions (e.g., Dryad, Mendeley). However, it does not explicitly state when not to use it or suggest alternatives beyond naming sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• relateA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description goes beyond the readOnlyHint annotation by detailing exactly what the tool does (detect shared accessions, cross-identifiers, explicit links, version lineage) and what it does not do (read file columns, fetch files, execute join/merge/conversion). No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, well-structured, and front-loaded with the core functionality. Every sentence adds necessary information without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of detecting relationships between resources, the description covers all essential aspects: purpose, input constraints, types of relationships detected, what is not done, and behavior on failures. An output schema exists, so return values need not be detailed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%. The description adds context that the ids are 'source-prefixed' and explains the purpose ('resource ids to relate'). It also reiterates the min and max items, adding value beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it returns 'metadata-level join/harmonization HINTS' for given resource ids, specifying the types of relationships detected (shared accessions, cross-identifiers, explicit links, version lineage) and what it does not do. This distinguishes it from siblings like list_sources, fetch, and operate.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description specifies the input range (2-10 ids) and advises to resolve ids first if only a search result is available. It also clarifies that per-id failures are reported and not fatal. While it doesn't explicitly state when not to use, the context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• searchA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses key behaviors: fan-out across sources, error reporting, ontology expansions, fallback behavior when LLM endpoints are missing, and pagination details. No contradiction with readOnlyHint annotation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded, but somewhat lengthy. Every sentence adds value, but could be slightly more concise. Still, no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers all aspects: purpose, usage, parameters, edge cases, and output. Given the complexity (17 parameters, multiple source integrations), the description is thorough and leaves no major gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, but the description adds significant value by explaining the semantic meaning of each parameter (e.g., how ontology expansions work, the effect of collapse_mirrors, and the interaction between rank and multi_query).

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: searching across multiple public research archives for various data types. It lists specific sources and result kinds, distinguishing it from sibling tools like resolve and fetch.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit guidance on when to use this tool (for searches) and when to use siblings (resolve for full records, fetch for downloads). Also explains usage of advanced parameters like collapse_mirrors, understand, and multi_query with conditions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• operateA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations declare readOnlyHint=true, and the description adds details: peek uses DuckDB SUMMARIZE, reads whole file, honors source-size ceiling; sql is read-only using view 'data'; failures documented. No contradiction.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is informative but slightly verbose; each sentence adds value but could be tightened. Front-loaded with purpose, then operations, then prerequisites.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 5 operations, 6 parameters, and no output schema, the description adequately covers all ops, parameter usage, prerequisites, failure modes, and links to other tools (resolve). Indirectly describes return types.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema covers 5/6 params (83%) with descriptions; the description adds meaning by explaining each op value, file addressing by catalog id + file name, default n=20, and sql view 'data'. Adds significant value beyond schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it inspects/queries remote tabular files without downloading, lists five specific operations (schema, preview, head, sql, peek), and distinguishes from siblings like fetch (download) and resolve (ID lookup).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says when to use (inspect/query without download), prerequisites (resolve id first, need [operate] extra), failure conditions (fails if not operable), and hints at alternatives via sibling context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.