Server Quality Checklist

Tool Scores

• get_observationsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It adds useful context: data is synthetic sandbox data, and responses are capped to protect against runaway responses. However, it does not explicitly state that the operation is read-only (though 'Get' implies it) or describe the output format.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the purpose, then key limitations. Every word earns its place; no fluff or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with 2 params and no output schema, the description covers the essential purpose and limitations (synthetic data, cap). It could optionally mention ordering or time range, but the examples and constraint 'recent' give a reasonable picture. Overall, sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, and both parameters have clear descriptions in the schema. The tool description adds context about 'recent' and 'capped,' but does not elaborate on parameter specifics. Baseline 3 is appropriate since the schema already documents the parameters well.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool fetches recent clinical observations (vitals, lab results) for a patient by id. It uses a specific verb and resource, and the examples distinguish it from sibling tools like get_patient and summarize_conditions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage: call this when you need a patient's recent observations. It does not explicitly state when to prefer this over siblings or mention any alternatives/exclusions, but the context of patientId and the tool name make the use case reasonably clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_patientA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses a key behavioral trait: 'Synthetic sandbox data only,' which informs the user about the non-production environment. The verb 'Get' also implies a read-only operation. It does not detail error behavior or return format, but for a simple get-by-id tool, the core behavior is sufficiently transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences, front-loaded with the primary purpose and followed by the important environmental note. Every word earns its place, with no unnecessary detail or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with one parameter, no output schema, and no nested objects, the description is complete. It states what it returns (demographic details), the input required (FHIR id), and the environment (synthetic sandbox). No additional context is needed for an agent to select and invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%: the parameter patientId is described as 'The FHIR Patient resource id.' The description also mentions 'by their FHIR id,' which reinforces the same meaning but does not add new semantic details beyond the schema. The schema already handles the parameter semantics adequately.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool 'Get demographic details for a single patient by their FHIR id.' The verb 'Get' is specific, the resource is 'patient', and the scope ('single patient', 'demographic details') distinguishes it from sibling tools like search_patients (which implies searching) and get_observations (which targets a different resource).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage: use this tool when you have a specific FHIR id and want demographic details for a single patient. However, it does not explicitly mention when not to use it or suggest alternatives like search_patients for searching without an id. The context is clear but lacks explicit exclusions or alternative references.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• summarize_conditionsA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It explicitly labels itself as a 'data-retrieval tool' (implying read-only) and adds the key disclaimer that it is not clinical or diagnostic, which discloses the tool's limitations. It does not detail return format or error handling, but these are less critical for a simple retrieval tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core purpose, and every clause adds context (nature of data, non-clinical disclaimer). There is no redundant or vague language.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with one parameter and no output schema, the description covers the essential aspects: what it does, that it's a retrieval, the sandbox-only context, and a caution about clinical use. The absence of output schema details is acceptable because the description's 'plain-language summary' provides the expected return nature.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema covers the single parameter patientId with a clear description ('The FHIR Patient resource id'). The tool description adds no additional parameter semantics beyond that, so the baseline score of 3 for 100% schema coverage applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Get' and the resource 'a patient's recorded conditions (diagnoses)', and specifies the output as a 'plain-language summary'. This distinguishes it from sibling tools like get_patient or get_observations, which focus on different resources and formats.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides important usage context: 'Synthetic sandbox data only' and 'not a clinical or diagnostic tool', which tells the agent when and why to use it. However, it does not explicitly name sibling tools or provide comparisons for when to choose one over another, so it stops short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_patientsA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses a key non-obvious behavior: returns synthetic test data only, never real patient records. However, it does not describe the return format or error behavior, but this is acceptable for a simple search tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three concise sentences, each providing distinct value: what it does, its synthetic data guarantee, and workflow guidance. It is front-loaded and free of redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple search tool with no output schema, the description covers the core purpose, data safety, and usage sequence. It implies the return includes patient ids but does not explicitly state the return structure. Given the low complexity, this is adequate, though a bit more detail would make it completely self-sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% for both parameters (name and limit), so the schema fully documents parameter semantics. The description adds no extra parameter details beyond what the schema already provides, earning the baseline score of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches for patients in the FHIR sandbox by name, specifying the resource (patients) and the action (search). It also explicitly distinguishes from siblings by saying to use it before get_patient or get_observations to find a patient's id.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit guidance: 'Use this first to find a patient's id before calling get_patient or get_observations.' This clearly indicates when to use this tool and the order relative to alternatives, plus notes it returns synthetic data only.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.