Server Quality Checklist

Tool Scores

• get_statsB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It hints at grouping behaviors ('count by year, conference, etc.') but does not state whether it is read-only, how results are formatted, if there are limits, or how parameters interact.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that communicates the core function efficiently. Every word contributes to the meaning, with no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no output schema and no annotations, the description provides a minimal but adequate overview of what stats are returned (counts by year, conference). Missing details such as the exact response structure or how multiple filters combine prevents a higher score.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema covers 100% of the parameters with descriptions for both tier and keyword. The description adds no additional semantics beyond saying 'search criteria', so it merely restates what the schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool gets statistics about papers, with examples of counts by year or conference. It distinguishes itself from sibling tools like search_papers by focusing on aggregates rather than individual results, though it does not explicitly name alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance is given on when to use this tool versus search_papers or get_conference_papers. The 'search criteria' hint implies usage when aggregated stats are needed, but there is no direct comparison or exclusion.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_conference_papersA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It discloses that the tool returns all papers for a specific conference-year, implying a read-only operation. However, it does not describe the return format, pagination, or any error behaviors. Since this is a getter, the lack of explicit read-only declaration is a minor gap, but the 'get' verb is informative.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single, front-loaded sentence with no wasted words. Every piece of information (action, resource, scoping) is included efficiently.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple two-parameter tool with no output schema, the description conveys the core behavior and required inputs. It lacks detail about the response structure, but the tool's purpose is straightforward. It is sufficient for selection and invocation, though not for predicting the exact output shape.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema provides descriptions for both parameters ('Conference key' and 'Conference year'), so coverage is 100%. The description adds only the context that the conference is a security conference, not the meaning of each enum value or year constraints. Thus it neither improves nor detracts from the schema's minimal parameter descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Get all papers') and the scope ('from a specific security conference in a specific year'), differentiating it from sibling tools like search_papers (which implies query-based searching) and list_conferences (which lists conferences). The mention of 'all papers' establishes that this is a bulk retrieval, not a filtered search.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not explicitly state when to use this tool versus alternatives. It implies usage for retrieving full conference proceedings, but provides no guidance on when to choose search_papers or get_stats instead. There is no mention of prerequisites like consulting list_conferences to get conference keys.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_papersA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of behavioral disclosure. It usefully notes 'keyword: Search in paper titles' and the available tier/conference filters, but omits details like result ordering, pagination, or response structure. This leaves moderate gaps but no contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description opens with a clear, direct purpose sentence and then organizes filters in a scannable bullet list. It is slightly longer than necessary due to schema duplication, but remains efficient and readable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 7-parameter search tool with no required fields and no output schema, the description covers the main filters and scoping. It omits the 'limit' parameter, but the schema fully documents it. Overall context is sufficient for an agent to invoke the tool effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters. The description adds minor value by clarifying 'keyword' is title-only and showing conference keys, but mostly reiterates schema information.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Search') and resource ('security research papers') and clearly distinguishes from siblings by listing the conferences covered. It also enumerates the filtering dimensions, providing a precise scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description states clear use cases: searching papers by keyword, author, year, conference, and tier. It does not explicitly mention when to prefer siblings like get_conference_papers or list_conferences, but the search-oriented context is unambiguous and sufficient.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_conferencesA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. The description is transparent about the read-only nature and the returned data, but it doesn't disclose any additional behavioral traits such as pagination or whether the list is exhaustive. For a simple list tool, this is adequate but not exceptional.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, focused sentence that conveys the purpose and output without any fluff. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter, simple list tool, the description fully specifies what the tool does and what it returns. No output schema is needed because the description explicitly mentions the fields ('keys and full names'). The tool is simple enough that this description is complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and the schema is empty, so there is nothing to explain. The description adds no parameter-specific meaning, but this is not a gap given the absence of parameters. Baseline for 0 params is 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('List') and the resource ('all supported security conferences'), and specifies the output ('their keys and full names'). This distinguishes it from sibling tools like search_papers or get_conference_papers, which focus on papers.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for obtaining the list of available conferences, which is distinct from the siblings' paper-focused functions. However, it doesn't explicitly state when to use it versus alternatives or provide exclusions, though the context makes it obvious.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.