Server Quality Checklist

Tool Scores

• get_paperA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses that the tool returns the complete paper payload and mentions the content (abstract, summaries, method/task entities). It implies read-only behavior via 'Fetch' and 'inspect', though it doesn't explicitly state lack of side effects. Sufficient for a simple read operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core action, and every clause adds value: what it fetches, the identifier, when to use, and what it returns. No fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple with one parameter and an output schema (though not shown), so the description doesn't need to detail return structure. It covers purpose, usage, parameter meaning, and return payload. The only gap is the parameter name mismatch, preventing a perfect score.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has one parameter `paper_id` with 0% description coverage, so the description must compensate. It does explain the identifier as `paper_uuid` and notes it comes from search/list results, adding meaning. However, the inconsistency between the description's `paper_uuid` and the schema's `paper_id` could confuse an agent, reducing clarity.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Fetch' and the resource 'a single ACL conference NLP paper', specifying the identifier as `paper_uuid` (Qdrant point id). It also distinguishes from siblings by mentioning it is used after obtaining an ID from search_papers or list_papers, making its purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit usage guidance: 'USE THIS to inspect a paper in full detail when you already have its ID from a search_papers or list_papers result.' This clearly indicates when to use and implicitly contrasts with searching or listing, offering context for tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_papersA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden. It discloses a key behavioral detail – pagination via `offset` defined as 'the last paper_uuid from the previous batch' – which goes beyond the schema. It does not mention ordering, result shape, or read-only nature, but those are either obvious or covered by the output schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, front-loaded with purpose, then usage, then parameter/pagination details. Every sentence adds distinct value with no fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description explains the core listing functionality and pagination sufficiently, and an output schema exists for return values. However, it leaves out some filter parameters (e.g., year_min/year_max) which are non-obvious from names alone, making it slightly incomplete for a 12-parameter tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It lists 'year/venue/acronym/field_of_study/author' and 'offset', covering 5 of 12 properties. It omits range filters (year_gt/lt/min/max), limit, collection_id, and doesn't clarify whether 'acronym' maps to collection_acronym. The offset semantics are well explained, but the partial coverage leaves gaps.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool 'Browse ACL conference NLP papers using structured filters only (no semantic query)' – a specific verb+resource+constraint. It distinguishes from siblings by explicitly excluding semantic queries and by giving concrete examples like 'all ACL 2024 papers'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives an explicit when-to-use: 'USE THIS to list papers when you know the filters but have no search text' and provides examples. The 'no semantic query' phrase implies that for search text, one should use a different tool (search_papers). This is nearly as clear as naming the alternative.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_papersA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It explains the embedding model, dimensions, similarity ranking, and the vector_name feature for searching different summary dimensions. It does not describe the output structure, but the output schema exists, and the search operation is inherently read-only.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single dense paragraph that leads with the core purpose and method, then adds filter options and vector_name details. It is longer than strictly necessary but every sentence contributes useful information without repetition or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (14 parameters) and the presence of an output schema, the description is remarkably complete. It covers the embedding mechanism, query semantics, structured filters, and vector_name customization, providing the agent with sufficient context to select and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It successfully explains the meaning of major filter parameters (year, venue, collection acronym/id, field of study, author) and vector_name. However, limit and min_score are not explicitly described beyond their inferred names, leaving a minor semantic gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies this as a semantic search tool for ACL conference papers, with a specific verb ('Semantic search'), a defined resource, and a distinctive method (embedding with Sentence-Transformers and cosine similarity). It distinguishes itself from siblings like list_papers and get_paper by emphasizing ranked relevance search.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use the tool: 'USE THIS when the user has a research topic or question and wants to find relevant papers.' This gives clear usage context, but it does not mention when to use alternatives (e.g., list_papers or get_paper), so it stops short of a full when/when-not comparison.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_collection_infoA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of behavioral disclosure. It does an above-average job by noting that distinct values are 'sampled' (not exhaustive) and lists exactly what metadata is returned. It does not explicitly state that the operation is read-only, but 'Discover' and the absence of parameters strongly imply this; the sampling caveat is especially valuable.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long, front-loads the core purpose, and packs relevant details (schema contents, sampled values, usage guidance) without filler. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (no parameters), the presence of an output schema, and the clear usage context, the description is complete. It explains what the tool returns, why it exists, and when to invoke it relative to sibling tools.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the parameter-semantics dimension has little to evaluate. Per the rubric, a zero-parameter tool receives a baseline of 4. The description adds no parameter-specific details because none exist, which is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb ('Discover') and clearly identifies the resource ('ACL NLP papers collection schema'), then enumerates what the tool reveals: vector dimensions, distance metrics, total paper count, and sampled distinct values for filterable fields. This differentiates it from sibling tools like search_papers and list_papers by positioning it as a schema-discovery tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'USE THIS first' and ties the tool to concrete scenarios: when you need to know venues, years, or fields before building search_papers or list_papers filters. This provides clear guidance on when to use it and implies that the sibling tools are the follow-up alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.