Server Quality Checklist

Tool Scores

• summarize_visitsA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It does not disclose whether the operation is read-only, destructive, requires authentication, or stores the summary. The word 'consolidate' leaves behavioral ambiguity.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with a parenthetical for detail, followed by a brief use-case sentence. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the moderate complexity (3 parameters, nested objects, no output schema, no annotations), the description lacks details on return value, error behavior, and the effect of optional patient data.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%; all parameters have descriptions. The tool description adds the overall purpose but does not provide additional semantic context beyond what the schema already offers.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool consolidates multiple past visits into a longitudinal summary with key findings and active diagnoses, naming specific sources (SOAP notes, free-text summaries) and use cases (care coordination, chart prep). It effectively distinguishes from sibling tools like list_specialties and get_note.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description mentions usefulness for care coordination and chart prep, implying context, but does not explicitly state when not to use the tool or mention alternatives among siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_audio_statusA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must fully disclose behavior. It only states it checks status, but does not mention that it's a read-only operation, possible status values, error cases, or side effects. Minimal transparency for a polling tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  One concise sentence (14 words) that front-loads the purpose. No redundancy or wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple status check tool with no output schema and no annotations, the description is incomplete: it does not hint at possible return values (e.g., 'completed', 'failed') or what happens if the noteId is invalid. Given the async nature, this information is important for an agent to handle responses correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The single parameter noteId is fully described in the input schema (100% coverage). The description adds 'by noteId' but does not provide additional meaning or format beyond what the schema already says.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the verb 'Check' and the resource 'processing status of an asynchronous audio-to-SOAP job', distinguishing it from sibling tools like transcribe_audio_to_soap which initiates the job. The tool name and description align well.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Indicates the tool is for checking status after an asynchronous job, but does not explicitly state when not to use it or mention alternatives. Could be more direct about using it after transcribe_audio_to_soap returns a noteId.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• transcribe_audio_to_soapA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, description bears full burden for behavioral traits. It discloses async processing, waiting behavior, and supported file formats. Missing details on error handling, file storage, or size limits. Adequate but not thorough.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with core action and key behaviors. No redundant information; every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 12 parameters, nested objects, and multiple optional features (include_* options, specialty, patient info), the description only covers audio input and output format. Does not mention customization options or output structure, leaving significant gaps for agent understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema covers 50% of parameters with descriptions. Description reinforces audio_path and async behavior (wait_for_completion, timeout_seconds) but does not explain other parameters like patient, template, or custom_instructions. Adds marginal value beyond schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the specific verb 'transcribe' and resource 'audio recording to SOAP note', and distinguishes it from siblings like 'list_specialties' and 'generate_soap_note' by focusing on audio input. The supported formats and behavior (sync/async) further clarify purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Description implies use when you have a local audio recording of a clinical encounter, and mentions short vs long recordings with async behavior. However, it does not explicitly state when not to use this tool (e.g., for transcript-only tasks) or list alternatives among siblings. Usage is implied but not exhaustive.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• generate_soap_noteA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It discloses that billing codes are AI decision support, but does not discuss other behavioral aspects like potential errors, need for review, or limitations of the AI-generated content. Provides some transparency but incomplete.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long, front-loads the primary purpose, and every sentence adds value. It is appropriately sized for the tool's complexity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 9 parameters (2 required) and no output schema, the description covers the main output (SOAP note structure) and optional features. It explains the specialty selection via reference to a sibling tool. Missing details on how patient or custom_instructions parameters affect the output, but overall sufficient for an agent to understand the tool's function and main inputs.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds context about optional outputs (billing codes, patient summary) but does not significantly expand on parameter meanings beyond what the schema provides. For example, 'patient' object description already says 'enrich the note.' Minimal added value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool generates a structured SOAP note from clinical transcript or shorthand, and lists optional additions like billing codes and patient summary. It effectively distinguishes from sibling 'transcribe_audio_to_soap' which handles audio input.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides a caution about billing codes requiring clinician review but lacks explicit guidance on when to use this tool vs alternatives like transcribe_audio_to_soap for audio input or list_specialties for available specialties. Usage context is implied but not directly addressed.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_noteA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description must cover behavior. It discloses auto-expiration (expires_at) and implies read-only retrieval, but does not mention error handling (e.g., what if noteId invalid or expired) or authentication requirements.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, front-loaded with action, no unnecessary words. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Lacks output schema, so description should hint at return structure. It mentions expires_at but not other fields like note content or metadata. Adequate but incomplete for a retrieval tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% and already describes noteId format and origin. Description adds minimal value beyond restating the parameter's role.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Retrieve' and the resource 'generated note', and distinguishes it from sibling tools by specifying it fetches results of async audio jobs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit context: use when status is 'completed' after an audio job. It implies when not to use (before completion) but does not explicitly state alternatives like get_audio_status.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_specialtiesA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description bears the full burden. It discloses that no API key is needed, which is a helpful behavioral note. However, it omits details like whether it is read-only, pagination limits, or rate limits. For a simple list tool with no parameters, this is adequate but not exhaustive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no fluff. It front-loads the action and immediately adds practical context. Every word serves a purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter tool with no output schema, the description provides everything needed: what it lists, how to use the output, and access requirements. It is fully complete given the tool's simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema is empty (0 parameters), and the schema description coverage is 100%. The description adds value by explaining that the output (specialty IDs) should be used as an argument for note generation, which is not in the schema. Given zero parameters, a baseline of 4 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description specifically states 'List the medical specialties (and template variants) supported by SOAPNoteAPI', clearly identifying the action and resource. It distinguishes from sibling tools (e.g., generating notes, transcription) by focusing on listing reference data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly instructs 'Use the returned specialty IDs as the `specialty` argument for note generation', providing direct guidance on when and how to use the tool. It also mentions 'No API key required', clarifying access requirements.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.