Server Quality Checklist

Tool Scores

• summarize_protocolA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description discloses model selection behavior (uses Claude Haiku if key is set, else deterministic template), which is helpful. However, it lacks information on error handling for invalid document IDs.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, front-loaded with purpose, no extraneous information. Every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter tool, the description adequately covers output structure and model behavior. Minor gap: no discussion of edge cases or required dependencies.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage on 'document_id', and the description does not elaborate on this parameter, leaving its type or format unclear.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Generate a structured summary of a single protocol' and lists specific outputs (phase, indication, etc.), distinguishing it from siblings like cluster_adverse_events or list_documents.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for summarizing a protocol but does not explicitly state when to use this tool versus siblings or provide any exclusion criteria.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• cluster_adverse_eventsA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description implies a read-only operation through 'Identify and group' and describes the output structure. It lacks explicit statements on safety (e.g., non-destructive) or side effects, which would raise the score. The parameter behavior (omission => all protocols) is well explained.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences with no unnecessary words. The first sentence states the primary action, the second details output and parameter behavior, making it efficient and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given one optional parameter and an existing output schema (not shown but referenced), the description is nearly complete. It explains output and parameter behavior. A minor gap is the lack of explicit mention that it operates on existing mentions without modification.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Despite 0% schema description coverage, the description adds significant meaning to the only parameter (`document_ids`) by explaining that omission clusters across all protocols. This compensates for the schema's lack of detail, though it could clarify what constitutes a document ID.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool groups adverse-event mentions by body system, with a specific verb ('Identify and group') and resource ('adverse-event mentions'). It distinguishes from siblings like `extract_entities` and `list_documents` by focusing on clustering adverse events across protocols.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides clear context for when to use the tool (across one or more protocols) and behavior for the optional parameter. However, it does not explicitly state when not to use it or mention alternatives like `summarize_protocol`, though the sibling names make the distinction inferable.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• extract_entitiesA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully discloses behavior: extraction, scope control via document_id, and result grouping capability. No destructive implications mentioned, but it is a read operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three concise sentences front-loading purpose, then usage detail. Every sentence adds value with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the single optional parameter and existence of output schema, the description covers main behavior and result structure. Minor omissions like pagination or limits do not detract significantly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Adds context about omitting document_id leading to all-document search, but does not explain its type or format. Schema coverage is 0%, so description partially compensates but leaves gaps.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Extract' and the resource 'clinical entities' with explicit types (drugs, conditions, interventions, endpoints, populations). It distinguishes from sibling tools like cluster, list, and summarize.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides clear context on when to omit document_id (run across all documents) and that each entity carries source document_id. Lacks explicit when-not or alternatives but offers sufficient guidance for typical use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_documentsA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It discloses the output structure (id, title, path, indication, phase) and that it lists all documents. It does not mention read-only nature or potential side effects, but for a list tool this is generally implicit. Slight gap but acceptable.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, each serving a clear purpose: define the action, detail the output, and provide usage context. No unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no parameters and an output schema, the description sufficiently explains what the tool does and what it returns. It could mention edge cases or error conditions, but for a straightforward list tool, it is largely complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0 parameters, so the baseline is 4. The description does not need to add parameter information, and it correctly omits any.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists all clinical trial protocol documents, specifies the return fields (id, title, path, indication, phase), and explicitly distinguishes from siblings by suggesting subsequent use of extract_entities or summarize_protocol.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description directly advises to use this tool first to discover available documents, then pass the id from its result to the sibling tools extract_entities or summarize_protocol, providing clear workflow guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.