Server Quality Checklist

Tool Scores

• watch_institutionA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the transparency burden and does well by disclosing the incremental behavior (only NEW flags, local state) and the `since` floor. It does not address output format or rate limits, but the output schema covers return values.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences deliver purpose, example, behavioral note, and parameter guidance without redundancy. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a two-parameter tool with an output schema and no annotations, the description covers the core scanning purpose, incremental behavior, and parameter hints. It lacks alternative-tool guidance but is otherwise complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description compensates by explaining `since` as a YYYY-MM-DD publication-date floor and providing a concrete example for `ror`. Both parameters receive meaningful context beyond schema fields.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool scans an institution's works by ROR ID and flags papers or papers citing flagged papers, using a specific verb and resource. It does not explicitly differentiate from sibling tools like check_references, so it misses the top score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'Repeat calls report only NEW flags' implies a monitoring use case, and the `since` parameter guidance is included. However, there is no explicit statement of when to use this tool versus alternatives such as check_references.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_referencesA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full transparency burden. It discloses that `paper_doi` causes references to be fetched from OpenAlex and checked, which is useful behavior. However, it does not mention potential network dependencies, error conditions, read-only nature, or what 'check' entails beyond the existence of an output schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no redundant information. It front-loads the core action ('Check a reference list') and then compactly enumerates the exact input options without elaboration.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description adequately covers the tool's purpose and input modes, and an output schema exists, so return values do not need to be described. It could be more complete with an explicit note about what 'checked' means or typical use cases, but for a three-parameter tool with clear input constraints, it is sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description is the sole source of parameter meaning. It fully defines each parameter: `dois` as a list of DOIs, `bibtex` as a .bib file's text, and `paper_doi` as a paper DOI whose cited references are processed. This adds complete semantic value beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Check a reference list') and the resource ('reference list'). It distinguishes the tool from siblings like get_editorial_status and watch_institution by focusing on reference checking, and further specifies the three accepted input modes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly instructs to provide exactly one of `dois`, `bibtex`, or `paper_doi`, which is essential usage guidance. It also explains the special behavior for `paper_doi` (OpenAlex fetch). It does not explicitly say when to prefer this tool over siblings, but the sibling purposes are clearly different, so this is a minor gap.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_editorial_statusA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses the output structure (source, evidence URL, date, confidence, conflicts) and possible status values, giving agents a clear picture of behavior. It does not mention error handling or authorization, but for a read-only lookup this is sufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  One sentence that is front-loaded with the core purpose, followed by specific statuses and result fields. No wasted words or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter) and the presence of an output schema, the description is complete enough. It covers the key aspects: purpose, possible outcomes, and returned fields, without over-explaining.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has a single 'doi' parameter with no description. The tool description only says 'one work', implying the DOI identifies the work, but does not provide format, examples, or additional guidance. With schema coverage at 0%, the description only partially compensates.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves the editorial status of a single work, listing specific status values (retracted, corrected, etc.). This distinguishes it from sibling tools like check_references and watch_institution, which serve different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use (when you need editorial status of a work) but does not explicitly state exclusions or alternatives. The scope is clear, but no direct comparison to sibling tools is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.