Server Quality Checklist

Tool Scores

• build_resourceA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses nesting behavior (CodeableConcepts, Dosage), return format {ok, resource} or {ok: false, issues}, and validation. No annotations provided, so description carries full burden. However, it's ambiguous whether the resource is persisted; could be clearer about side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, front-loaded with purpose and key guidance, no wasted words. Efficient and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers purpose, return format, supported types, and a critical rule about codes. However, with 16 parameters and no output schema, the description lacks detailed parameter mapping and resource-type-specific instructions, leaving gaps for correct usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has 16 parameters with 0% coverage, but description only mentions resource_type and code usage rule. Does not explain parameters like onset, value, unit, etc., or how they map to resource types. The description adds minimal value over the schema for parameter meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool builds and validates a FHIR resource from flat fields, lists supported resource types, and contrasts with writing FHIR JSON manually, distinguishing it from siblings like validate_resource and lookup_code.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides clear guidance: 'Call this instead of writing FHIR JSON by hand' and 'Use a code returned by lookup_code, never an invented one.' Also lists supported resource types. Missing explicit when-not-to-use or comparison to siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_resourcesA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided. Description indicates it is a read operation returning full FHIR JSON, and implies prerequisite of a loaded bundle. Lacks details on permissions, errors, or other behavioral traits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, no fluff, front-loaded with core action and differentiator.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given low complexity (1 param, no output schema), description covers purpose, usage, and return format. Missing parameter details and behavioral transparency, but overall sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Single parameter resource_type has no description in schema (0% coverage). Description fails to add any meaning, e.g., valid values or examples.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states it gets all resources of a type as full FHIR JSON, and distinguishes from sibling get_coded_entries by noting it carries fields the flattened view doesn't.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says when to call this tool ('when you need fields...') and implies alternative (get_coded_entries). Does not provide when-not or other alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• validate_resourceA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so description must cover behavioral traits. It states 'Never raises' (no exceptions) and mentions return format (issues[] with severity/diagnostics/expression). However, it does not disclose authentication needs, rate limits, or whether it is read-only.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two tightly written sentences: first defines purpose, second gives usage and output structure. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description explains return value (issues[]). It is sufficient for a validation tool with simple output. Sibling context suggests no further explanation needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so description must compensate. It explains the 'resource' parameter subtly by referring to a 'FHIR resource (JSON object)', but does not detail the optional 'resource_type' parameter. Minimal added value over schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states verb 'Validate', resource 'FHIR resource', and scope 'against the spec, including required value-set bindings'. This distinguishes it from sibling tools like build_resource or get_resources.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Gives explicit when to use: 'Call this after constructing or editing FHIR JSON yourself'. Also instructs what to do with output: 'read issues[] and correct the resource'. Missing explicit when not to use or alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_coded_entriesA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully discloses that this is a read operation that returns flat records with specific fields and optionally filters by status. It does not mention destructive actions, authentication, or rate limits, but for a read-only tool the disclosure is sufficient. The behavior is transparent for typical use.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the main purpose and followed by usage guidance. Every sentence provides value with no redundancy. Ideal conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations or output schema, the description covers the tool's purpose, typical use case, return format, and optional filtering. It lacks details on edge cases (e.g., invalid resource type, empty results) but is complete enough for primary use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate. It explains that resource_type is the type of resource (suggesting conditions, medications, etc.) and that status is an optional filter with example 'active'. This adds meaning beyond the bare schema, but does not fully clarify valid resource types or constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool reads coded identity of every resource of a type in the loaded bundle as flat records with specific fields. It specifies the resource types (conditions, medications, allergies, observations) and that it resolves medicationReference. This is specific and distinct from sibling tools like get_resources or resolve_reference, though not explicitly compared.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description advises to call this first for questions about conditions, medications, allergies, or observations, providing clear usage context. It also notes that it resolves medicationReference automatically. However, it does not explicitly state when not to use it or mention alternative tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• lookup_codeA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It mentions that results are ranked 'tightest-match-first', which is a useful behavioral trait. However, it does not disclose auth requirements, rate limits, result limits, or error handling. The description is adequate but not comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise: three sentences covering purpose, usage guidance, and ranking/system restriction. Every sentence adds value with no fluff. Front-loaded with the core action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema and no annotations, the description covers the essential points: what it does, when to use it, ranking, and optional parameter. It could be more complete by mentioning result format or what to do if no match, but it is sufficient for a simple lookup tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It provides concrete examples for 'query' and explains the 'system' parameter as 'optionally restrict to one code system URI'. This adds meaning beyond the schema, though it does not elaborate on what a code system URI looks like.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'search' and the resource 'terminology catalog for codes' with examples. It distinguishes itself from inventing codes by explicitly saying 'never invent or recall one', and the sibling 'list_codes' is a different operation (listing all vs. free-text search).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Call this whenever you need a code' and provides a caution against inventing codes. It also mentions optional system restriction. However, it does not explicitly exclude scenarios or compare with sibling tools like 'list_codes' or 'get_coded_entries', so it loses one point.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• resolve_referenceA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It discloses the behavior for contained references but does not mention error handling, what happens if the reference is not found, or any other side effects. Adequate but not comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no redundancy. The first sentence states the primary purpose, and the second covers a specific use case. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity and lack of output schema or annotations, the description covers the main use cases (absolute and contained references). It does not cover error scenarios or edge cases, but for a reference resolution tool it is reasonably complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must add meaning. It explains the 'reference' parameter with example formats ('Patient/123', 'urn:uuid:...') and clarifies that 'parent_resource_id' is needed for contained references. This adds significant value beyond parameter names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Resolve', the resource type (FHIR reference), and the context ('within the loaded bundle'). It distinguishes between absolute references and contained references, making the tool's purpose precise and well-defined.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains when to use the tool and provides specific guidance for contained references (parent_resource_id). However, it does not explicitly exclude cases when not to use it or mention alternative tools from the sibling list.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_codesA
  4.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, description carries full burden. Reveals it lists all codes with optional filter, but doesn't mention pagination, error handling for invalid system URIs, or return format. Minimal but not misleading.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences: first states purpose and filtering, second provides usage guidance and alternative. Front-loaded and efficient with no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list tool with one optional parameter and no output schema, description covers core purpose and usage. Lacks details on output format or pagination, but sufficient for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, but description explains the only parameter 'system' as a filter by code system URI. Single parameter is well-clarified, adding meaning beyond schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states verb 'list' and resource 'codes in local terminology catalog', with optional filter by system URI. Distinguishes from sibling 'lookup_code' by contrasting browse vs search.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says when to call this tool ('to browse what codes are available') and when to use the alternative ('use lookup_code to search by name'). Provides concrete use case example.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• load_bundleA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so the description carries full burden. It explains error behavior with invalid bundles returning issues[] and gives an example. However, it does not mention overwriting previous bundle or idempotency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, no fluff. First sentence states purpose, second adds usage order and error handling. Every sentence is valuable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description adequately covers purpose, usage order, and error behavior. It could clarify the concept of 'working bundle' or return value, but overall is complete enough.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, but the description adds meaning by stating 'from a file path or JSON string', clarifying the source parameter's two possible formats. This compensates well.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool loads a FHIR Bundle from a file path or JSON string and sets it as the working bundle for read tools. This distinguishes it from sibling tools like build_resource or get_resources.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Call this first when given bundle data', giving clear usage context. It does not list when not to use it or alternatives, but the context is sufficient.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.