Server Quality Checklist

Tool Scores

• updateA
  3.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description must fully disclose behavioral traits. It mentions the tool mutates a request (write operation), but does not address destructive effects (e.g., cancel), authorization needs, rate limits, or side effects. This is insufficient for a mutation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the main purpose and uses bullet points for modes, making it easy to scan. Every sentence adds value without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers mode-specific parameters adequately but lacks information about return values (despite having an output schema), error conditions, or safety warnings. For a tool with 8 parameters and 5 modes, more high-level guidance is needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description compensates by explaining which parameters are required for each mode (e.g., choice for confirm, doi for repoint). However, the id parameter is not described, leaving ambiguity.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Mutate a request' and enumerates specific modes (confirm, repoint, flag, priority, cancel), each with a distinct purpose. This distinguishes it from sibling tools (status, submit, submit_file) which handle different operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage by detailing each mode and required parameters, but does not explicitly state when to use this tool versus alternatives. It provides clear context for usage but lacks exclusion guidelines.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• statusA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It discloses that the tool returns a single card or list, but does not mention side effects, authentication, or other behavioral traits. For a read tool, this is adequate but not comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very concise with a clear structure using 'Args' and 'Returns' sections. Every sentence adds value with no unnecessary information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the low complexity and existence of an output schema, the description covers the basics: input parameters and return types. Missing details like error handling or pagination, but these are not critical for this simple read tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema coverage is 0%, so the description fully documents both parameters: 'id' as a UUID string and 'filter' with supported keys. This adds significant meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states 'Read one or many paper requests,' which clearly indicates a read operation on a specific resource. This distinguishes it from sibling tools like 'submit' and 'update', which are write operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explains when to use 'id' versus 'filter' and that 'id' overrides 'filter'. However, it does not explicitly mention when not to use this tool or direct users to alternatives for write operations.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• submitA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, description fully discloses behavior: resolution, dedup, queuing, idempotency, statuses, and per-agent rate-limiting. No contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured with Args and Returns sections. Concise yet comprehensive; every sentence provides value. Front-loaded with purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers all parameters and return values, including idempotency and statuses. Output schema exists but description still adds context. Complete for tool complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Despite 0% schema coverage, description thoroughly explains each parameter: ref (subsets, raw), dry_run, source, priority, created_by. Adds significant meaning beyond bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states 'Submit a paper request' and explains the process (resolves metadata, checks dedup, queues). Distinguishes from siblings like submit_file by focusing on reference-based submission.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides implicit guidelines through dry_run parameter for testing and priority/created_by for rate-limiting. Also warns not to fabricate quotes from non-ingested papers. Lacks explicit comparison to siblings but still informative.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• submit_fileA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description fully bears the burden. It discloses behavioral traits such as re-opening specific states ('failed', 'extract_failed', 'needs_user'), refusing to overwrite closed/cancelled requests, and the return value being the request record flipped to 'ingesting'.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a concise intro, 'Args' section, 'Returns' section, and a final constraint note. Every sentence adds value, with no redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 6 parameters, no annotations, and an output schema, the description is complete. It explains parameter usage, return value, and even the background process. No critical information is missing for an AI to use the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description provides detailed semantics for each parameter, including mutual exclusivity (url vs content_base64, request_id vs ref), fallback behavior for filename, and usage of created_by. This fully compensates for the lack of schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Attach a user-supplied PDF to a paper request.' It provides specific use cases (user-dropped PDF or pre-loaded file) and distinguishes from siblings by focusing on PDF attachment, not request creation or status.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use the tool (e.g., 'when a user drops a PDF...') and explains the two parameter groups (url vs content_base64, request_id vs ref). It lacks explicit when-not-to-use guidance but provides clear context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.