Server Quality Checklist

Tool Scores

• search_papersC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are present, so the description carries the full burden. It only states 'Search' without revealing behavioral traits such as whether it is read-only, pagination behavior, authentication requirements, or result ordering. The description adds minimal transparency beyond the basic action.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that conveys the core purpose efficiently. There is no wasted content, though it could be slightly more structured with additional key behaviors.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 4 parameters (with 0% schema coverage) and an output schema (not visible), the description is too brief. It lacks information on pagination, result format, sorting, and filtering behavior. Sibling tools exist but are not mentioned, leaving gaps in completenss.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description must compensate but only mentions searchable fields ('title, abstract and keywords'). It does not explain the meaning of 'limit', 'category', 'discipline', or how filters combine. The schema property titles are self-explanatory, but the description adds no deeper semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Search'), the resource ('CSL papers'), and the search fields ('by title, abstract and keywords'). It distinguishes from siblings like 'get_paper' (which retrieves a specific paper) and 'list_categories'/'list_disciplines' (which list metadata).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not provide guidance on when to use this tool versus alternatives. It neither lists exclusions nor explains scenarios where siblings would be more appropriate. The usage context is implied but not explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_disciplinesC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It discloses it lists data with counts and optional filtering, but no mention of read-only nature, error conditions, or side effects. For a simple list operation, the minimal description is insufficient for full transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence that is front-loaded with the core action and result. No verbose or redundant text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has an output schema, so return values need not be described. However, for a list operation with optional filtering, the description lacks details on pagination, sorting, or what 'counts' means. It is minimally complete but could benefit from additional context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so description must compensate. It adds that 'category' is an optional filter, but does not explain valid values, format, or constraints like null handling. With one parameter, this is inadequate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'List' and the resource 'CSL disciplines', and specifies it returns 'counts'. The tool name 'list_disciplines' reinforces this. While it distinguishes from siblings like 'get_paper' and 'search_papers', the acronym 'CSL' is assumed known.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus alternatives like 'list_categories' or 'get_dataset_statistics'. The only usage hint is 'Optionally filter by category', which implies use cases but does not compare to other tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_paperB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden for behavioral disclosure. It only states 'Get', implying read-only behavior but lacks details on error handling, rate limits, or any side effects. The presence of an output schema is not leveraged in description.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single sentence with no extraneous words. However, it could be more informative without sacrificing conciseness. It is front-loaded but leaves gaps.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with one parameter and an output schema, the description is functional but minimal. It does not mention what the tool returns (e.g., a paper object) or acknowledge the output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, requiring description to compensate. The description merely says 'by id' without explaining the paper_id format, length, or provenance. It adds no meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Get one CSL paper by id', specifying the verb, resource, and unique identifier. This distinguishes it from sibling tool search_papers which retrieves multiple papers.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool vs alternatives. While the context implies single-paper retrieval by ID, the description does not mention that search_papers is for queries or provide usage conditions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_dataset_statisticsA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It implies a read operation ('get') and mentions the data source (local SQLite), but does not disclose performance characteristics, authorization needs, or side effects. For a simple retrieval tool, this is adequate but minimal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that immediately conveys the tool's purpose. No unnecessary words or details are present. It is front-loaded and earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the output schema exists, the description is not required to explain return values. However, the description is very brief and could hint at the type of statistics (e.g., counts, distributions). For a zero-parameter tool, this is passable but not fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters exist, so schema coverage is 100%. The description need not add parameter details. The baseline for zero-parameter tools is 4, and the description meets that standard by not introducing confusion.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves 'CSL dataset statistics' from the 'local SQLite database', providing a specific verb and resource. It distinguishes from sibling tools (get_paper, list_categories, etc.) which focus on individual entities rather than dataset-level statistics.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is given on when to use this tool versus alternatives. The description only states what it does, without mentioning prerequisites, context, or when not to use it. This forces the agent to infer usage from the tool's purpose alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_categoriesA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided. Description implies a read-only operation listing categories with counts, but does not disclose behaviors like caching, pagination, or authorization requirements. Minimal transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  One concise sentence with no extraneous information. Every word contributes to the purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given zero parameters and an existing output schema (though not shown), the description is nearly complete. Minor gap: no mention of response structure, but output schema covers that.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters exist, and schema coverage is 100% trivially. Description adds value by specifying that results include counts, which is beyond what the empty schema provides. Baseline 4 appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Specific verb 'List' with resource 'CSL categories' and qualifier 'with counts'. Clearly distinguishes from sibling tools like list_disciplines, as it targets categories specifically and provides count information.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool vs alternatives. Usage is implied from the name and purpose, but no when-not or alternative tools are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.