Server Quality Checklist

Tool Scores

• fhir_readB
  3.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It does disclose that a 404 means genuinely not found, which is valuable error semantics. It also indicates the response is FHIR R4 format. However, it does not mention authentication requirements, rate limits, or what happens on other errors (e.g., 401, 500). For a read-only GET operation, this is above minimum but not rich.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no wasted words. The URL template is front-loaded, followed by a critical error-handling note and a cross-reference to search. Every sentence earns its place, making it highly efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple single-resource read, the description covers the URL construction, 404 semantics, and resource type allowlist. It does not explicitly state the tool's overall purpose (fetching one resource) or mention authentication, which might be assumed from a prior auth step. Given no output schema, it could also clarify the response shape, though 'FHIR R4' implies a FHIR resource. Slight gaps remain in purpose clarity and auth context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. The URL template '{iss}/{resourceType}/{id}' gives meaning and ordering to the three parameters, and the allowlist note constrains resourceType. However, it does not explain what iss represents (likely the server base URL) or any format expectations for id or resourceType beyond being strings. This is partial compensation for the lack of schema documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states 'GET {iss}/{resourceType}/{id} as FHIR R4', which clearly indicates a read operation targeting a specific FHIR resource. It also clarifies the 404 semantics, which helps distinguish it from a search that might return empty results. However, it does not explicitly say 'retrieve a single resource by ID', relying on the URL pattern to imply it, and it does not contrast with fhir_search directly beyond the allowlist note.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no explicit guidance on when to use this tool versus fhir_search or other siblings. It only mentions 'Same resourceType allowlist as search', which hints at shared constraints but does not explain the core difference (single resource lookup vs query). A clear statement like 'Use this to fetch one resource by ID; use fhir_search for queries' is missing.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• fhir_auth_statusA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds the behavioral note 'Never prints token or PEM', which is a valuable safety disclosure not available in annotations (since none are provided). It also lists exactly what will be reported. However, it does not state whether the operation is read-only, whether it can fail (e.g., if discovery hasn't been run), or what the response format will be. Given zero annotations, the description carries the burden but provides only partial behavioral transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, tightly written sentence that front-loads the verb 'Report' and then enumerates the status fields. It ends with a concise safety disclaimer. There is no fluff or repetition; every word earns its place. This is an exemplary model of conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter tool with no output schema, the description lists all the status values it reports and adds a safety constraint. It lacks details on the exact output structure (e.g., JSON keys vs. plain text) and does not specify whether the tool requires prior setup (like discovery), but these are minor given the tool's simplicity. Overall, it provides enough context for an agent to call and interpret the results reasonably.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters (schema is an empty object), so the baseline for parameter semantics is 4 according to the rubric. The description correctly mentions the fields that will be reported, which are effectively the tool's 'output' rather than input parameters. Since there are no parameters to document, the description adds what little meaning is needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the verb 'Report' and lists specific status fields (mode, ISS, R4 lock, etc.), making the tool's purpose clear: to report FHIR authentication status. It is distinguishable from siblings like fhir_search and fhir_read by focusing on auth state rather than data operations, and from smart_discover by being a status check. However, it does not explicitly differentiate itself from siblings in the text.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus the sibling tools. It doesn't mention scenarios like 'before calling fhir_search' or 'when you need to verify authentication state'. The purpose is implicit from the name and list of status fields, but no explicit usage context is given, and no exclusions or alternatives are referenced.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• smart_discoverA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It includes one genuinely useful behavioral note ('Does not invent endpoints'), signaling the tool only returns what the server actually advertises. However, it omits other relevant behavior such as error handling when iss is not allowlisted, whether authentication is needed, or any required scopes, leaving the behavioral profile only partially disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three terse, information-dense sentences with no filler. The endpoint is front-loaded, followed by the spec version, the parameter constraint, and the single most important behavioral caveat. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-optional-parameter discovery tool with no output schema, the description covers the endpoint, the constraint, and the key behavior. Return format is not described, but SMART discovery responses are standardized by the cited spec, reducing the need. Only edge-case behavior (unallowlisted iss handling) is left unspecified, which is minor for this tool's simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate. It does add meaning by substituting iss into the URL path and by stating the v1-allowlist constraint. However, it never explicitly defines iss as the FHIR server base URL or describes the valid format/value domain beyond the allowlist, leaving some semantic burden on inference from the '{iss}' placeholder.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific verb (GET) plus a concrete resource (.well-known/smart-configuration) with a spec version (SMART App Launch 2.2.0). The '{iss}' prefix clarifies it targets a specific FHIR server's discovery document, and it is clearly distinct from the sibling tools (fhir_auth_status, fhir_search, fhir_read) which handle auth status and searches/reads rather than capability discovery.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides a useful precondition (iss must be v1-allowlisted) and clarifies the call is optional-parameter. However, it gives no explicit guidance on when to prefer smart_discover versus sibling tools like fhir_auth_status, nor any exclusion criteria. The 'when to use vs alternatives' context is implied by its distinct purpose but never stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• fhir_searchA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden and does well: it declares a GET request, response type (searchset), resourceType allowlist, _count limits, single-page behavior, and explicitly states it does not create resources ('does not invent Patients'). Auth and error handling are absent, but the key read-only and pagination traits are disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise, packing essential details into five short sentences. It front-loads the HTTP method and resource pattern, then lists constraints without filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given three parameters including a nested object, zero schema coverage, and no output schema, the description supplies key facts (resourceType allowlist, _count limits, single page, read-only), but omits the meaning of 'iss' and the syntax for search parameters in 'params'. These gaps leave an agent without full construction guidance.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate. It explains resourceType via the allowlist and gives _count default/max, but does not define 'iss' (presumably the FHIR base URL) or the general structure/keys of the 'params' object. Meaning is added for resourceType and partially for _count, but iss and general params remain undocumented.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'GET {iss}/{resourceType} as FHIR R4 searchset' with a specific verb and resource, and lists the allowlisted resource types. This distinguishes it from sibling fhir_read (single resource read) and other tools by implying search vs discovery/auth.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies a search/query use case via HTTP GET and resourceType allowlist, but does not explicitly name alternatives or when-not-to-use it. The context suggests fhir_read is for single resources, but no explicit routing is provided; usage is implied rather than stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.