Server Quality Checklist

Tool Scores

• fhir_searchB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description alone must disclose behavioral traits. It only states the return type but omits whether the operation is read-only, what happens on no results, rate limits, or authentication requirements. This is insufficient for a search operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise with two short sentences. Every word is necessary and front-loaded. No redundant or vague language.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema and no annotations, the description could have provided more context about return values (e.g., bundle structure) or usage examples. However, the schema is rich and fully covers parameters, partially compensating. The description is minimally complete for a straightforward search tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The tool description adds no extra meaning beyond the schema—it does not explain the 'params' object format or provide examples. The description is adequate but does not enhance parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Search a FHIR R4 resource type' with a specific verb and resource. It also specifies the return type as 'FHIR Bundle (searchset)'. This distinguishes it from sibling tools like fhir_read (single resource) and fhir_search_next (pagination).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like fhir_read or fhir_search_next. It fails to mention use cases, exclusions, or prerequisites such as required authentication or server capabilities.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• fhir_readA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so description carries full burden. It states it returns full JSON but omits details like error handling (e.g., missing resource) or authorization requirements.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two efficient sentences with no fluff. Front-loaded with action and scope.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Adequate for a simple read operation with full schema coverage. Could mention behavior on not found, but not critical given simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has 100% description coverage for both parameters, but descriptions are minimal. Overall description adds little beyond schema (reiterates 'by type and logical ID'). Baseline 3 appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states it reads a single FHIR R4 resource by type and ID and returns full JSON. Verb 'Read' and resource specification distinguish it from siblings (search, capabilities, validate).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Implies use when reading individual resources by type+ID, but does not explicitly mention alternatives like fhir_search for multiple resources or fhir_capabilities for metadata.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• fhir_search_nextA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It states the return type (FHIR Bundle) and the presence of '_next_url' for further pages, but omits other behavioral traits like read-only nature or potential errors. However, the behavior is straightforward for a pagination tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very concise: two sentences that front-load the action and provide essential details without any fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one required parameter, no output schema, no nested objects), the description covers all necessary aspects: input, behavior, and output format.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The single parameter 'next_url' is fully described in the schema (100% coverage). The description adds value by clarifying its origin ('from a previous fhir_search Bundle'), but this is largely redundant with the schema description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the tool's purpose: following a Bundle pagination link returned by fhir_search. It distinguishes itself from siblings like fhir_search and fhir_read by focusing on pagination.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description specifies when to use the tool (after a previous fhir_search result) and what input to provide ('_next_url'). It implicitly excludes use without a prior search, but does not explicitly state when not to use or list alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• fhir_capabilitiesA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description fully discloses behavior: it retrieves a read-only summary of the CapabilityStatement. It does not mention side effects, which are absent, and states it should be called first, implying safety.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two efficient sentences that front-load the purpose and include a usage directive. Every sentence adds value with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (no parameters, no output schema), the description fully covers what it does and when to use it. Sibling tools are contextually present but not detailed, which is appropriate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters exist, so baseline 4 applies. The description does not need to add parameter meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Retrieve' and the resource 'FHIR server's CapabilityStatement' along with specific contents (version, software, resource types, search parameters). It differentiates from siblings which are for reading, searching, and validation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says 'Call this first to understand what the server supports,' providing clear when-to-use guidance. It omits explicit when-not-to-use statements, but the context suffices.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• validate_against_profileA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden and adequately discloses that it validates using a specific validator, returns conformance status, error count, and issue details, which informs the agent about the tool's behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences, front-loaded with the primary action, and no unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description adequately explains the return value (conformance status, error count, issue details), and covers the purpose for all parameters. The tool is fully described for an agent to use correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so baseline is 3. The description adds value by specifying supported profiles (US Core, IPS) and the return information, which goes beyond the parameter descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool validates a FHIR R4 resource against a StructureDefinition profile using the HAPI validator, and distinguishes from sibling tools (fhir_capabilities, fhir_read, etc.) which handle other operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use the tool (for validation against US Core, IPS, or base FHIR profiles), but does not explicitly state when not to use it or mention alternatives beyond the sibling list.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.