Server Quality Checklist

Tool Scores

• get_userA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses that it retrieves the 'authenticated user's' profile, implying authentication requirements, and mentions 'demographic and contact details' as output content. However, it lacks details on error conditions, rate limits, or whether it's read-only/safe—critical for a tool with no annotations. This is adequate but has clear gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences with zero waste: the first states the purpose, and the second provides usage guidelines. It's front-loaded with the core action and efficiently adds context without redundancy. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (simple retrieval), no annotations, no output schema, and 0 parameters, the description is minimally adequate. It covers the purpose and usage but lacks behavioral details like response format or error handling. For a tool with no structured fields to rely on, it should do more to be complete, but it meets the minimum viable threshold.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters, and schema description coverage is 100% (empty schema). The description doesn't need to explain parameters, and it correctly avoids adding unnecessary details. A baseline of 4 is appropriate for zero-parameter tools, as there's no parameter semantics to cover beyond what the schema provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Retrieves the authenticated user's FHIR profile' (specific verb+resource). It adds context about accessing 'demographic and contact details,' which helps distinguish it from generic 'get' operations. However, it doesn't explicitly differentiate from sibling tools like 'read' or 'search,' which might also retrieve user data, preventing a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear usage guidance: 'Use this tool when you need to access the current user's demographic and contact details.' This gives context for when to invoke it. However, it doesn't specify when NOT to use it or mention alternatives among sibling tools (e.g., 'read' might retrieve other users' profiles), so it falls short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• createA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and does well by disclosing key behavioral traits: it specifies that servers may reject resources for profile violations or mandatory binding issues, which is crucial for a write operation. However, it doesn't mention other potential behaviors like rate limits, authentication needs, or response formats.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, with the first sentence stating the core purpose. The second sentence adds required input details, and the third provides usage context and behavioral notes. Every sentence earns its place, though it could be slightly more streamlined.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a mutation tool with 4 parameters, no annotations, and no output schema, the description is adequate but has gaps. It covers the purpose and some behavioral aspects (server rejections), but lacks details on prerequisites, error handling, or return values. Given the complexity, it should do more to compensate for missing structured data.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description adds minimal value beyond the schema by mentioning 'full resource payload in JSON form' and 'specified type,' but it doesn't provide additional semantic context or usage examples for parameters. Baseline 3 is appropriate given high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Executes a FHIR `create` interaction'), the resource involved ('new resource of the specified type'), and distinguishes it from siblings by specifying it's for adding new data rather than reading, updating, or deleting. The examples (Patient, Observation) further clarify the purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool ('when you need to add new data'), but it doesn't explicitly mention when not to use it or name specific alternatives among the sibling tools (e.g., use 'update' for existing resources, 'read' for retrieval). The guidance is helpful but lacks sibling differentiation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• readA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It explains the tool's purpose and constraints (single resource retrieval, not for bulk queries) but lacks details on error handling, rate limits, authentication requirements, or response format. It does mention that search parameters must match those from 'get_capabilities', adding some context, but overall behavioral traits are minimally covered.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficiently structured in two sentences: the first states the purpose and scope, and the second provides usage guidelines and parameter context. It's front-loaded with key information and avoids redundancy, though it could be slightly more concise by integrating the parameter details more seamlessly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (4 parameters, nested objects, no output schema, and no annotations), the description is adequate but has gaps. It covers purpose and usage well but lacks details on behavioral aspects like error handling or response format. Without annotations or an output schema, the description doesn't fully compensate for these missing elements, leaving some context incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description adds marginal value by explaining that search parameters 'refine the response' and operations are 'custom FHIR operations', but it doesn't provide additional syntax or format details beyond what the schema specifies. This meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool performs a FHIR 'read' interaction to retrieve a single resource instance by type and ID, distinguishing it from bulk query tools like 'search' and other CRUD operations like 'create', 'update', and 'delete'. It specifies the exact verb ('retrieve'), resource ('single resource instance'), and scope ('by its type and resource ID').

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool ('when you know the exact resource ID and require that one resource') and when not to use it ('do not use it for bulk queries'). It also mentions alternatives implicitly by contrasting with bulk queries, though it doesn't name specific sibling tools like 'search' directly. The inclusion of 'If additional query-level parameters or operations are needed...' further clarifies usage scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• deleteA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden and does well. It discloses that this is a destructive operation ('remove a single resource'), explains conditional behavior ('deleting the resource only if it matches the given criteria'), mentions server validation ('must match an existing resource instance'), and describes the return type ('FHIR OperationOutcome'). It doesn't cover rate limits or authentication needs.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded. The first sentence states the core purpose, followed by specific usage scenarios and parameter interactions. Every sentence adds value with no wasted words, and the structure logically progresses from general to specific details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a complex destructive operation with 4 parameters, no annotations, and no output schema, the description does well. It explains the tool's behavior, parameter interactions, and return type. However, it doesn't mention prerequisites like authentication or potential side effects, which would be helpful given the destructive nature.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description adds some semantic context by explaining how parameters interact (e.g., 'If you include searchParam, the server will perform a conditional delete'), but doesn't provide significant additional meaning beyond what's in the schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Execute a FHIR `delete` interaction on a specific resource instance.' It specifies the verb ('delete'), resource ('FHIR resource'), and distinguishes from siblings like 'create', 'read', 'update', and 'search' by focusing on removal operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool: 'when you need to remove a single resource identified by its logical ID or optionally filtered by search parameters.' It explains conditional deletion and operation execution scenarios. However, it doesn't explicitly state when not to use it or name alternatives among siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_capabilitiesA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes the tool's purpose and constraints (e.g., it's for metadata discovery, not resource fetching), but lacks details on potential errors, rate limits, or authentication requirements. However, it adds significant context beyond basic functionality.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose, followed by usage rules and exclusions in two concise sentences. Every sentence earns its place by providing essential guidance without redundancy, making it highly efficient and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (metadata discovery for FHIR operations) and lack of annotations or output schema, the description is largely complete. It explains the tool's role and constraints clearly, though it could benefit from mentioning what the output includes (e.g., search parameters, operations) or error handling, which would enhance completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, clearly documenting the single required parameter 'type' with examples. The description adds no additional parameter-specific information beyond what the schema provides, such as format details or constraints, so it meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Retrieves metadata') and resource ('FHIR resource type'), distinguishing it from sibling tools like 'read' or 'search' that fetch actual resources. It explicitly contrasts with those operations by stating 'Do not use this tool to fetch actual resources.'

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool ('MUST always be invoked before performing any resource operation') and when not to use it ('Do not use this tool to fetch actual resources'). It implicitly positions this as a prerequisite for other operations like search, read, create, update, or delete, offering clear alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• searchA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes key behaviors: it's a read-only operation (implied by 'search' and contrast with create/update/delete), may return paginated results for large datasets, and returns a bundle or list. However, it doesn't specify authentication requirements, rate limits, or error handling, leaving some gaps for a tool with no annotation support.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose in the first sentence, followed by usage guidelines and behavioral notes in subsequent sentences. Each sentence adds clear value: defining the tool, specifying when to use it, warning against misuse, and noting pagination behavior. There is no redundant or wasted text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description does well to cover purpose, usage, and key behaviors like pagination. However, it lacks details on return values (bundle/list structure), error cases, or server-specific constraints, which would be helpful for a search tool with complex parameters. It's mostly complete but has minor gaps in output and error context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already fully documents the two parameters (type and searchParam). The description adds minimal semantic value beyond the schema—it mentions 'resource type' and 'search-parameters' but doesn't provide additional context like format examples or constraints not in the schema. This meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Executes a standard FHIR `search` interaction on a given resource type, returning a bundle or list of matching resources.' It specifies the verb ('search'), resource type ('FHIR resource'), and distinguishes it from siblings like create, update, delete, and read by explicitly mentioning it's for querying multiple resources based on search parameters.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool ('when you need to query for multiple resources based on one or more search-parameters') and when not to use it ('Do not use this tool for create, update, or delete operations'). It also implies an alternative for single-resource retrieval (likely 'read') by contrasting with querying multiple resources, though it doesn't name specific siblings like 'read' directly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• updateA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively explains that this is a destructive write operation ('replacing an existing resource instance's content'), mentions that the payload must include all required fields and any data to preserve, describes optional conditional updates and custom operations, and notes the return value includes either the updated resource or error details. It doesn't cover rate limits or authentication requirements, but provides substantial operational context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized at three sentences, front-loaded with the core purpose, followed by usage guidance, and ending with return value information. Every sentence adds value, though it could be slightly more concise by combining some concepts about optional parameters.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a complex mutation tool with 5 parameters, no annotations, and no output schema, the description does well by explaining the destructive nature, parameter interactions, and return values. It covers the essential context needed to use the tool correctly, though could benefit from mentioning authentication or error handling specifics.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema description coverage, the baseline is 3. The description adds meaningful context beyond the schema by explaining the purpose of each parameter: 'type' identifies the FHIR resource, 'id' is the logical ID, 'payload' must contain complete resource data for replacement, 'searchParam' enables conditional updates, and 'operation' allows custom operations. This provides practical guidance on how parameters work together.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool performs a FHIR 'update' interaction by replacing an existing resource's content with a provided payload. It specifies the verb ('replacing'), resource ('FHIR resource instance'), and distinguishes it from siblings by mentioning it's for overwriting entire resources when you know the logical ID, unlike create (new resources) or read/search (retrieval operations).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use this tool ('when you need to overwrite a resource's data in its entirety, such as correcting or completing a record, and you already know the resource's logical id') and mentions conditional updates and custom operations as optional use cases. It distinguishes from siblings by implying this is for full replacements rather than partial updates or other operations.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.