Server Quality Checklist

Tool Scores

• get_fhir_configB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool returns configuration settings, but lacks details on permissions needed, rate limits, error handling, or whether it's a read-only operation. For a tool with no annotation coverage, this is a significant gap in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is brief and front-loaded, stating the purpose in the first sentence and the return in the second. There's no wasted text, but the structure could be slightly improved by integrating the return statement more seamlessly or adding minimal context.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 0 parameters, 100% schema coverage, and an output schema exists, the description is adequate but minimal. It explains what the tool does and what it returns, but lacks context on usage, behavioral traits, or how it fits with siblings, making it incomplete for optimal agent understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0 parameters with 100% coverage, so the schema fully documents the lack of inputs. The description doesn't need to add parameter details, but it implicitly confirms no parameters are required by not mentioning any. This meets the baseline for zero-parameter tools, though it could briefly note the absence of inputs for clarity.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('Get') and resource ('current FHIR server configuration'), making it immediately understandable. However, it doesn't differentiate this tool from its siblings (like 'read_fhir_resource' or 'search_fhir_resources'), which might also retrieve configuration-related data, so it doesn't reach the highest score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention any context, prerequisites, or exclusions, such as whether this is for administrative settings versus patient data access, leaving the agent to infer usage from tool names alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_fhir_resourcesB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions the return type ('A FHIR Bundle containing matching resources or an error message') and a specific use case for 'custom_headers' in Zus servers, but does not cover critical aspects like authentication requirements, rate limits, error handling details, or whether this is a read-only operation. This leaves significant gaps for an agent to understand behavioral traits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, starting with the core purpose. It uses a structured format with 'Args:' and 'Returns:' sections, making it easy to parse. While efficient, the 'custom_headers' explanation is slightly verbose but adds necessary context, so it earns its place without significant waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (3 parameters, 0% schema coverage, no annotations, but with an output schema), the description is partially complete. It covers parameter basics and return values, but lacks details on authentication, error scenarios, and behavioral constraints. The output schema likely handles return structure, reducing the need for description there, but overall gaps remain for safe and effective tool invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It adds meaning by explaining each parameter: 'resource_type' with examples ('Patient', 'Observation'), 'search_params' as an optional dictionary with examples, and 'custom_headers' with a specific use case. However, it does not fully detail all possible values, constraints, or formats beyond basic examples, leaving some ambiguity for the agent.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Search for FHIR resources using query parameters.' It specifies the verb ('search') and resource ('FHIR resources'), making it understandable. However, it does not explicitly differentiate from sibling tools like 'read_fhir_resource' or 'write_fhir_resource', which could involve similar resources but different operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage through examples (e.g., searching for 'Patient' or 'Observation'), but does not explicitly state when to use this tool versus alternatives like 'read_fhir_resource' for specific resource retrieval. It provides context for 'custom_headers' in multi-tenant scenarios, but lacks clear guidance on exclusions or prerequisites for general use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• write_fhir_resourceA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and does well by detailing steps like validation, endpoint determination, and error handling. It discloses behavioral traits such as validation using fhir-validator and handling of errors, which are beyond basic functionality. However, it misses some details like rate limits or authentication requirements, preventing a perfect score.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with bullet points and sections for Args and Returns, making it easy to scan. It's appropriately sized with no redundant information, but the bullet points could be more concise, and some sentences are slightly verbose (e.g., 'Returns validation errors or server errors for correction').

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (write operation with validation and custom headers), no annotations, and an output schema present, the description is mostly complete. It covers purpose, steps, parameters, and returns, but could benefit from more details on error types or success conditions, especially since the output schema exists but isn't described in the text.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It adds meaning by explaining that 'resource' is a FHIR resource as a JSON object and 'custom_headers' is optional with a specific example for Zus servers. This provides practical context beyond the schema's basic types, though it could elaborate more on resource structure or header constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool writes a FHIR resource to a FHIR server, specifying the action (write) and resource (FHIR resource). It distinguishes from sibling tools like read_fhir_resource and search_fhir_resources by focusing on creation/update. However, it doesn't explicitly differentiate from potential siblings like update_fhir_resource or create_fhir_resource if they existed, keeping it at 4 instead of 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for writing FHIR resources, with a note about Zus servers for multi-tenant access, suggesting context-specific application. However, it lacks explicit guidance on when to use this tool versus alternatives (e.g., no mention of when to use POST vs. PUT or how it differs from sibling tools like read_fhir_resource). The guidance is present but not comprehensive.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• read_fhir_resourceA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses the tool reads resources (non-destructive) and mentions error handling ('or an error message'), but lacks details on authentication needs, rate limits, or server-specific behaviors beyond the Zus headers example. It adds some context but is incomplete for a mutation-free read operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a clear purpose statement, parameter explanations, and return information. It uses bullet-like formatting under 'Args:' and 'Returns:' for readability. Some minor verbosity exists (e.g., repeating 'FHIR' could be trimmed), but overall it's efficient and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has an output schema (returns JSON), the description needn't detail return values. It covers the core purpose, parameters, and a key usage note (Zus headers). For a read operation with no annotations, it provides adequate context, though it could benefit from more behavioral details like error types or access requirements.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It explains all three parameters: 'resource_type' with examples ('Patient', 'Observation'), 'resource_id' as the target ID, and 'custom_headers' with a specific use case for Zus servers. This adds meaningful semantics beyond the bare schema, though it doesn't cover all possible header formats or constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Read a FHIR resource') with the target ('by type and ID from the FHIR server'), distinguishing it from sibling tools like 'search_fhir_resources' (which searches) and 'write_fhir_resource' (which writes). The verb+resource combination is precise and unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for retrieving specific resources by type and ID, but does not explicitly state when to use this versus alternatives like 'search_fhir_resources' for broader queries or 'get_patient_zus_upid' for patient-specific IDs. No exclusions or prerequisites are mentioned, leaving usage context somewhat open-ended.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_patient_zus_upidA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses that the tool searches for a Patient and extracts the UPID, and mentions it returns 'an error message if not found.' However, it lacks details on authentication needs, rate limits, or what specific error messages might be returned, leaving behavioral gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded with the core purpose, followed by context, parameters, and returns. Every sentence adds value: the first states the purpose, the second provides context, the third explains the search logic, and the last two detail parameters and returns, with no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (3 parameters, no annotations, but with an output schema), the description is fairly complete. It covers purpose, context, parameters, and return behavior. However, it could improve by addressing authentication or error specifics, though the output schema may handle return values.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It adds meaning by explaining each parameter's purpose: 'first_name: Patient's first name', 'last_name: Patient's last name', and 'builder_id: Optional Zus builder ID (string) to filter the search.' This clarifies semantics beyond the bare schema, though it doesn't detail format constraints or examples.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Get the Zus UPID (Universal Patient ID) for a Patient resource from Zus FHIR server.' It specifies the exact action (get), resource (Zus UPID), and distinguishes it from siblings by focusing on extracting UPIDs rather than general FHIR operations like read_fhir_resource or search_fhir_resources.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool: 'This is a Zus-specific tool for working with Zus Health's FHIR API' and 'Searches for a Patient by first and last name.' It implies usage for Zus-specific UPID extraction but doesn't explicitly state when not to use it or name alternatives among siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.