Server Quality Checklist

Tool Scores

• auditBibliographyA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds substantial operational detail beyond the readOnly/idempotent/openWorld annotations: 1-based indexing warnings, the shared index space between entries and parseErrors ('report parseErrors as UNCHECKED, never as clean'), truncation at 25 with excess dropped, the warning never to sum summary.retracted with verdict counts, per-entry error leniency, and anonymous-vs-key auth/rate-limit differences. No statement contradicts the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long (~470 words) but front-loaded: purpose, usage triggers, and input contract come first, with return and edge semantics after. The inline return-type spec and the 'Reading the result' section are verbose, but they are justified by the absence of an output schema; the structure is logical and each section earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema and a complex batch tool (two mutually exclusive input modes, truncation, parse errors, a retraction axis separate from verdicts, and auth gating), the description carries the full burden and covers nearly everything: input contract, return shape, 1-based indexing, sourceKey resolution semantics, error handling, scope limits, and rate limits. Nothing an agent needs to call it correctly is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all five parameters thoroughly (bibliography vs claims exclusivity, maxItems 25, checks default of ['retraction'], format enum, screenWithLlm default false). The description mostly restates the EITHER/OR input contract and the cap, adding only the cross-tool auth-gating link for screenWithLlm, so it stays at the baseline of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The first sentence states a specific action and resource: 'Verify a WHOLE bibliography in one call', and immediately differentiates it from its sibling: 'the batch counterpart to verifyCitation'. It also names the exact fabrication check (identifier resolving to a title that does NOT match) plus retraction lookup, so an agent can tell exactly what the tool does without opening the schema.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  An explicit 'Use when...' sentence lists concrete triggers: pasting a reference list, a .bib/.ris file, or phrasing like 'check all these citations at once' / 'audit my bibliography' / 'which of these references are fake or retracted'. It also states an explicit exclusion — 'it does NOT check whether a source supports the claim it is cited for' — which prevents confident misuse, and positions the tool as the batch counterpart of verifyCitation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• checkOpenAccessA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, idempotentHint, and destructiveHint, so safety is covered. The description adds substantial behavioral context beyond annotations: identifier resolution to DOI, the doi=null/reason='no_doi' failure mode, rejection of batches, full return shape, rate limits, API key options, and server-side Unpaywall caching.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long but densely informative; each sentence earns its place. It is front-loaded with purpose and usage triggers before diving into resolution behavior, return structure, and API details. Minor redundancy with the schema's parameter description prevents a perfect score.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity and absence of an output schema, the description covers everything: return format, error handling, accepted identifier types, rate limits, auth setup, sibling differentiation, and idempotency. An agent has enough information to select, call, and interpret results correctly without further lookup.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already has 100% coverage with a detailed description of the id parameter. The description adds further meaning by explaining how non-DOI identifiers are resolved, how resolution failures surface in the response, and how the return object relates to the parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource ('Check whether a single scholarly work is openly accessible') and further clarifies it finds the best legal version. It also explicitly differentiates from siblings by stating 'No sibling tool overlaps this — resolveIdentifier returns metadata but not OA status.'

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives concrete trigger phrases ('is this open access?', 'where can I read this for free?') and when to use it (wanting OA license/version before reuse). It also explicitly names resolveIdentifier as the non-overlapping alternative, providing clear when-to-use and when-not-to-use guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• formatCitationA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already carry readOnlyHint, idempotentHint, and destructiveHint=false; the description confirms these ('Read-only and idempotent — safe to retry') and adds substantial context beyond them: vancouver defaulting with an explicit ask-before-defaulting policy, alias resolution reporting styleUsed, batch processing in one round trip, the metadata block appended as a second text content item, rate-limit tiering, and API-key auth requirements (SCHOLAR_API_KEY, RAPIDAPI_KEY). No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long (~230 words) but dense — nearly every sentence carries operational information an agent needs (defaults, auth, return shape, sibling routing). It is front-loaded with purpose and usage before diving into details. Slight redundancy exists (identifier formats partially echoed in the schema's text parameter description), but no filler sentences.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite no output schema, the description fully specifies the return shape ('one of { text, html, items } depending on the output parameter, followed by a metadata block... appended as a second text content item') and tells the agent to surface it for reproducibility. It covers auth setup, rate-limit behavior, batch semantics, defaults, and all sibling tool boundaries — nothing an agent needs to call this correctly is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3; the description adds value on top by disclosing default behavior (vancouver if unspecified), the disambiguation instruction when style is ambiguous, alias resolution semantics ('harvard' → 'harvard-cite-them-right'), and the output default of text. It also cross-references resolveIdentifier for identifier formats instead of redundantly repeating them, which is efficient enrichment rather than restatement.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Opens with a specific verb+resource+outcome: 'Format scholarly identifiers into a finished citation in a specific style.' It then explicitly distinguishes itself from siblings — resolveIdentifier (raw metadata) and exportCitation (downloadable bibliography file) — so an agent can tell them apart without opening schemas.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  States exactly when to use it ('Use when the user wants a paste-ready citation string for a manuscript, slide, message, footnote, or in-line reference') and names the alternatives with their selection conditions ('Use resolveIdentifier instead when the user wants raw metadata to inspect or transform; use exportCitation when they want a downloadable bibliography file'). It even adds an ask-the-user rule for ambiguous style variants, which is actionable usage guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• verifyCitationA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, idempotentHint, and openWorldHint, and the description reinforces them ('Read-only and idempotent — safe to retry'). It goes well beyond the annotations by disclosing authentication gating (400 LLM_SCREEN_FORBIDDEN for anonymous callers), API key requirements and rate-limit effects, stage semantics for screenWithLlm, and the full return contract. No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Every sentence earns its place — purpose, pattern, triggers, constraints, gated behavior, return shape, verdict semantics, sibling exclusion, auth — but the whole thing is one dense, unbroken paragraph. The density is justified for a tool this complex, but breaking the return contract and verdict semantics into their own section would improve scannability.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema, the description carries the full burden of explaining return values — and it delivers: complete return shape, four verdict definitions with examples, error behavior for the gated LLM path, and sibling routing. Nothing an agent needs to call this tool correctly is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with rich per-parameter descriptions, so the baseline is 3. The description adds genuine value above the schema: the 'exactly one identifier' constraint that the schema's required array does not express, and how optional parameters modulate the verdict (author mismatch raises suspicion, >=2-year gap lowers confidence, container is a soft signal). It slightly overlaps the schema on identifier priority order, so not a 5.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Opens with a precise verb+object pairing — 'Verify a claimed citation against the resolved record at its identifier' — and immediately names the exact fabrication pattern it detects (real identifier + mismatched title, Topaz et al.). It explicitly differentiates itself from the overlapping sibling resolveIdentifier, so an agent can select it without opening either schema.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Gives explicit trigger conditions with example user phrasings ('is this real?' / 'check this DOI'), states the single-citation-per-call and exactly-one-identifier constraints, and explicitly names the alternative (resolveIdentifier) and why it is not appropriate. This is exemplary routing guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• checkRetractionA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnly, idempotent, and non-destructive behavior, and the description adds substantial context beyond that: intentional rejection of batch inputs, server-side DOI resolution, ISBN always returning null, Crossref `updated-by` source, rate-limit tiers, and the exact return shape. No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long but densely packed with necessary information: purpose, usage context, input restrictions, data source, output shape, sibling differentiation, and auth/rate limits. Each sentence serves a distinct purpose and the most critical usage guidance is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema, the description fully documents the return object structure. It also covers edge cases (no DOI, ISBN), authentication options, rate limits, and server-side behavior. Nothing an agent needs to invoke this tool correctly is omitted.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already documents the single `id` parameter at 100% coverage with a rich description. The tool description reinforces this by explaining why the parameter must be a single identifier and how resolution works. This adds value by clarifying the intended usage pattern (loop for multiple identifiers), going slightly beyond the bare schema baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Check whether a single scholarly work has been retracted, corrected, or had an expression of concern raised.' It names the sibling it does not overlap with ('resolveIdentifier returns metadata but not retraction status'), clearly distinguishing this tool from alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit when-to-use guidance is given: direct retraction queries and verification before citing in clinical/regulatory/evidence-synthesis contexts. It also states how to handle multi-paper audits by looping one call per identifier, and explains why batch input is rejected. No alternative is left ambiguous.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• resolveIdentifierA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint and idempotentHint, and the description reinforces them with 'Read-only and idempotent — safe to retry.' It then goes beyond the annotations by disclosing authentication options (anonymous free tier, SCHOLAR_API_KEY, RAPIDAPI_KEY), rate-limit behavior, and server-side caching surfaced via x-scholar-cache header. This is rich behavioral context the agent would not otherwise know.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long but tightly packed: every sentence carries distinct information—purpose, usage triggers, accepted formats, output shape, alternatives, safety, auth, and rate limits. It is front-loaded with the most decision-relevant facts and avoids filler or tautology.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having no output schema, the description fully specifies the return format (JSON array of CSL items with listed fields). It also covers identifier variants, prefixes, batch handling, auth, rate limiting, and sibling routing. Nothing an agent needs to invoke this tool correctly is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the schema already documents the text parameter thoroughly. The description adds value with concrete conversion examples (PMID → PMCID, arXiv → DOI, ISBN → CSL JSON) and clarifies that a batch resolves in one round trip, which aids interpretation beyond the schema's formal listing.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Resolve scholarly identifiers to structured CSL JSON metadata.' It enumerates accepted identifier types and names the two most relevant sibling alternatives (formatCitation, exportCitation) to disambiguate. An agent can immediately understand what the tool does and how it differs from the other rendering-oriented siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit when-to-use guidance is front-loaded: 'Use when the user wants raw bibliographic data to inspect, transform, or feed into another tool — not a formatted citation.' It also states when not to use it, naming formatCitation and exportCitation as alternatives. The batch behavior and single-round-trip claim give operational clarity.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• exportCitationA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations provide readOnlyHint and idempotentHint, and the description reinforces these while adding extra context: one round trip per batch call, return shape {content, format}, anonymous vs. keyed rate limits, and the fact that 'txt' is the only format that uses style. This goes well beyond what annotations alone convey.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long but every sentence earns its place. It is front-loaded with purpose, then usage, alternatives, behavior, and auth details in a logical order. The format list is dense but necessary to disambiguate otherwise cryptic enum values.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 4 parameters, no output schema, and a broad format enum, the description covers everything an agent needs to invoke it correctly: return shape, format-specific behavior, identifier handling, batch semantics, auth/rate limits, and clear routing relative to sibling tools. Nothing critical is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so baseline is 3. The description significantly adds semantics by explaining identifier prefix tolerance, batch separators (comma/newline), shortDOI expansion, the exact behavior of each format enum, and the interaction between style, lang, and format. This exceeds what the schema alone provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Export scholarly identifiers to a bibliography file format...'. It explicitly lists the target use cases (Zotero, Mendeley, EndNote, etc.) and names sibling tools (formatCitation, resolveIdentifier) that serve different needs, leaving no ambiguity about what this tool does.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear when-to-use guidance ('Use when the user wants a file...'), specifies which formats map to which reference managers, and explicitly contrasts with alternatives ('Use formatCitation instead...'). It also covers batching, API key tiers, and rate-limit behavior, so the agent knows exactly when and how to select this tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.