Server Quality Checklist

Tool Scores

• search_publicationsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It only states the search criteria (title and abstract keywords) but does not disclose behaviors like result limits, pagination, ordering, or side effects. Minimal beyond basic purpose.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single sentence that is efficient and to the point. No fluff, but it could be slightly more detailed without losing conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the basic action and search fields. With an output schema existing, the return structure is handled. However, lacking usage guidelines and behavioral traits, it feels minimally adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must clarify parameters. It adds that the 'query' parameter searches by title and abstract keywords, which provides some meaning beyond the raw schema. However, no format or operators are specified.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it searches synthetic publications by title and abstract keywords. It distinguishes from sibling tools like search_protocols, though it does not explicitly contrast them. The verb 'search' and resource 'publications' are specific.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool vs alternatives like search_protocols. No context about prerequisites, typical use cases, or scenarios where other tools might be preferred.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compute_aaa_statisticsB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must disclose behavioral traits. It only says 'compute simple descriptive statistics,' implying a read-only operation, but does not mention side effects, data sensitivity, or any constraints. The description is insufficient for full transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence that is front-loaded with the key action and target. Every word earns its place with no superfluous content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of parameters and existence of an output schema, the description is somewhat adequate but could be improved by specifying what 'simple descriptive statistics' includes (e.g., mean, median, count). It does not mention the output schema, which the instructions say is not required, but more detail would enhance completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are no parameters, so the schema coverage is complete. The description does not need to add parameter details, and it correctly implies the tool has no inputs. The baseline of 4 is appropriate here.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it computes descriptive statistics for the synthetic AAA cohort, using a specific verb and resource. However, it lacks specificity about which statistics are included, which slightly reduces clarity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus sibling tools. While siblings have different purposes (finding patients, metadata, protocols, publications), explicit usage context is missing.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_protocolsB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of disclosing behavioral traits. It states the tool searches and returns excerpts, implying a read operation, but fails to mention whether it is read-only, any required permissions, rate limits, or other important behaviors.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence of 10 words, front-loading the verb and resource. Every word contributes to the purpose, with no redundancy or irrelevant details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter, output schema exists), the description is minimally adequate. It explains the core function but omits details like whether results are paginated, how many excerpts are returned, or any filtering capabilities. It covers the basics but leaves gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has one parameter (query) with 0% description coverage in the schema. The description adds minimal meaning by indicating the query is used to search markdown files, but it does not explain query syntax, supported operators, or how excerpts are matched. The description partially compensates but remains insufficient.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Search markdown research protocol files and return relevant excerpts.' It specifies the verb (Search), the resource (markdown research protocol files), and the output (relevant excerpts). This distinctively separates it from sibling tools like compute_aaa_statistics or find_aaa_patients.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no explicit guidance on when to use this tool versus alternatives. While it implies it's for searching protocol files, it does not state when not to use it or suggest other tools for different contexts. The sibling tools list includes search_publications, but no differentiation is made.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_patient_metadataB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses that the data is synthetic and that the tool returns metadata for one patient/study, implying a read-only operation. However, it omits details like error handling, authentication needs, or rate limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the purpose without any extraneous content. Every word contributes to the overall understanding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter, output schema exists), the description covers the core purpose and basic behavior. However, it lacks usage context, such as when to use this tool versus searching for patients, and does not address potential error cases or expected outcomes.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 0%, and the description does not add meaning to the required 'patient_id' parameter beyond the schema's title. It only implies the parameter's role through the phrase 'for one patient/study', but lacks format, constraints, or examples.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns synthetic demographics and CT metadata for a single patient or study, using the verb 'Return' and specifying the resource and scope. This distinguishes it from sibling tools like find_aaa_patients (which finds patients) and compute_aaa_statistics (which computes statistics).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives, nor are there any prerequisites or exclusions. The description only states what the tool does without contextualizing its usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_aaa_patientsA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It does not disclose behavior like pagination, sorting, error handling, or performance characteristics. Only states basic query operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence of 12 words, no redundancy. Information is front-loaded and efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Adequate for a simple query tool with output schema handling return values. However, lacks context on use cases, limitations, or data scope. Could mention if it supports large datasets or timeouts.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, but description adds meaning by explaining 'AAA diameter threshold' for min_diameter_cm and 'optional age filter' for min_age. This compensates for lack of schema descriptions, though could clarify units (cm, years).

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states verb 'Find', resource 'patients', and specific criteria: AAA diameter threshold and optional age filter. This distinguishes it from sibling tools like compute_aaa_statistics and get_patient_metadata.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives. No mention of when-not or prerequisites. The description only tells what it does, not how to choose it over siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.