Server Quality Checklist

Tool Scores

• replication_statusA
  3.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden and excels by detailing live data retrieval, author-agnostic signers, returned fields (validation status, CiTO relation, repository, links, agreement pattern, claims, title/abstract), and behavior for unreplicated DOIs ('open').

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is detailed and somewhat verbose, but effectively structured with clear bullet points outlining return values. It could be more concise, but the length is justified by the breadth of information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema and one undocumented parameter, the description compensates by thoroughly explaining return values, data source, and behavior (e.g., 'open' for unreplicated). It is nearly complete for an agent to understand the tool's functionality.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The sole parameter 'doi' has 0% schema description coverage, and the description does not explain its format or meaning beyond the name. The context implies it's a DOI string, but explicit guidance is missing.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: to check if a DOI has been independently replicated and whether it held. It specifies the data source (Science Live verdicts from nanopub network) and distinguishes it from other systems like OpenAIRE Graph.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for obtaining replication status but does not explicitly state when to use this tool versus siblings (find_independent_software, radar, verified_claims). No when-not-to-use or alternative guidance is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_independent_softwareA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so the description carries the full burden. It discloses ranking by reuse signal, result fields (stars, rank_score), and the author-disjoint filter. It also clarifies it's not citations-based. Doesn't mention side effects, but as a query tool that's acceptable.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single dense paragraph with dashes and parentheticals. It front-loads the purpose but could be more concise. All sentences add value, so it's not wasteful, just stylistically heavy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema, it discloses key result fields (stars, rank_score). It covers input constraints and purpose. For a tool with 3 simple paramsaineit is reasonably complete, though the `limit` parameter's behavior is not described.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description explains `doi` (looks up authors) and `topic` (should be short and broad). `limit` is not explained, though its default 8 implies a count cap. Since schema coverage is 0%, the description partially compensates but not fully.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: finding reusable, author-disjoint software for replicating a paper's claim, ranked by reuse signals. It distinguishes itself from citation-based tools and siblings like find_dataset by focusing on code engines. The verb 'find' is explicit and the scope is well-defined.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides concrete usage guidance: pass the paper's DOI (authors are looked up) and a short topic, with advice to keep topic broad. It doesn't explicitly contrast with alternatives like find_dataset, but the context makes the use case clear. No when-not-to-use instructions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• radarA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses output details: papers with flags OPEN/VERIFIED, abstract (markup stripped), and independent reusable tooling. It also mentions impact-ranking. Since no annotations are provided, the description adequately covers behavioral traits, though it does not mention side effects or auth requirements.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with three sentences, each adding value. No redundant words. It is front-loaded with the main purpose and then elaborates efficiently.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with two parameters and no output schema, the description explains what is returned and gives usage guidance. It lacks explicit details on limit parameter behavior and the nature of the 'independent reusable tooling,' but is largely complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, so the description must add meaning. It provides usage advice for `topic` (keep short, terms AND-ed) but does not explicitly define `limit` beyond its default. The description adds partial value but leaves some parameter semantics implicit.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it returns 'Impact-ranked replication targets in a research field,' specifying that it returns high-impact OpenAIRE papers with OPEN/VERIFIED flags, abstracts, and tooling. This distinguishes it from siblings like find_independent_software and replication_status.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides clear advice: 'Keep `topic` short (2-3 words); OpenAIRE free-text terms are AND-ed.' This guides on parameter usage. However, it does not explicitly state when to use this tool versus alternatives or when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_datasetA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses that the tool does NOT search itself and returns instructions instead. It also clarifies the delegation to OpenAIRE. This is transparent about its behavior and limitations, though it could mention any side effects or prerequisites (none apparent).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is multi-sentence but each sentence adds value: stating purpose, clarifying the handoff, and providing behavioral instructions. It is front-loaded with the core purpose and remains focused. Slightly verbose with the 'Do not answer' part, but still concise enough.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity, the description is complete. It explains what the tool does, when to use it, how to handle the user request, and even mentions DOI citation. No output schema exists, but the description covers the necessary context for an agent to act correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description does not explain the 'topic' parameter at all. The schema provides no description either, and coverage is 0%. The parameter is optional with a default, but the description adds no meaning regarding how it is used or whether it matters. This is a clear gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's role: it provides instructions for finding a citable dataset, not the actual search. It explicitly says 'Where to find a DATASET' and differentiates from the OpenAIRE MCP. This is specific and distinguishes it from sibling tools like find_independent_software.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly says when to call it: 'Call this when the user asks the Radar to find data/a dataset'. It also mentions the alternative (OpenAIRE MCP) and provides a directive: 'Do not answer I can't search datasets — hand off to the OpenAIRE MCP'. This is excellent guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• verified_claimsA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description provides useful behavioral context: it is a read-only list operation that returns DOIs with verdicts and replication counts, and is author-agnostic and network-wide. It does not mention any destructive or complex behaviors, which is acceptable for a simple list tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with an additional sentence detailing output fields, front-loading the key action and scope without any unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given zero parameters, no output schema, and simple context, the description fully explains the tool's purpose and output format, leaving no gaps for an agent to misinterpret.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are no parameters, so the description does not need to add parameter context. The baseline score for zero-parameter tools is 4, and the description appropriately omits parameter details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'List every claim' and specifies the resource: claims from the nanopub network with Science Live replication verdicts, distinguishing it from sibling tools like radar and replication_status.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for obtaining the verified-knowledge corpus but lacks explicit guidance on when to use this tool versus alternatives, such as when to choose replication_status instead.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• replication_templateA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and does so thoroughly. It discloses that the tool returns a template repo link, scaffold contents, workflow steps, repo-name suggestions, availability checks, a quickstart command, and a handoff file. It also transparently notes that `gh repo create` copies a blank template, so the handoff is necessary—no surprising side effects are hidden.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a long single paragraph, but nearly every sentence adds useful information about behavior, outputs, or parameter effects. It front-loads the core purpose and sibling distinction, then expands into workflow and parameter semantics. A bulleted structure would improve scannability, but the density is justified given the tool's complexity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is comprehensive: it covers the tool's role in the larger loop, what it returns, how to use optional parameters, the end-to-end replication workflow, and why the handoff is necessary. It also supplies the template repo URL and names the nanopublication chain components. Given the absence of an output schema, this description sufficiently equips an agent to use the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must explain all parameters. It does explain doi/topic as name-suggestion inputs, owner as the GitHub org/user for availability checks, and dataset/software as inputs that trigger a handoff. It misses small format details (e.g., DOI syntax or whether software is a name or URL), but overall it adds substantial meaning beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool provides the FORRT replication template, scaffold details, and end-to-end workflow for producing a replication nanopublication chain. It explicitly differentiates itself from sibling discovery tools ('radar / replication_status / find_independent_software only DISCOVER') and says when to call it ('whenever the user wants to start, scaffold, or set up a replication').

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit when-to-use guidance: 'Call this whenever the user wants to start, scaffold, or set up a replication, or asks for "the FORRT replication template".' It also distinguishes this tool from sibling discovery tools and explains the context of the 'produce' half of the loop. It further describes how to adapt usage based on optional inputs like doi/topic, owner, and dataset/software.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.