Server Quality Checklist

Tool Scores

• get_transcriptC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states 'Retrieve,' implying a read-only operation, but lacks details on permissions, rate limits, error handling, or response format. This is a significant gap for a tool with no annotation coverage, making it minimally informative about behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's purpose without unnecessary words. It is front-loaded and wastes no space, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description is incomplete. It fails to address key aspects like what the retrieval returns, error conditions, or behavioral traits. For a tool with this complexity and lack of structured data, the description should provide more context to be fully helpful.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, with the parameter 'transcriptId' fully documented in the schema. The description adds no additional meaning beyond what the schema provides, such as format examples or context about transcript IDs. This meets the baseline score of 3, as the schema handles the parameter documentation adequately.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Retrieve') and resource ('transcription results by transcript ID'), making the purpose immediately understandable. It doesn't specifically differentiate from sibling tools like 'submit_transcription' or 'transcribe_file', which have different purposes, but the distinction is reasonably implied rather than explicitly stated.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. The description does not mention prerequisites, such as needing a transcript ID from a previous operation, or specify contexts where it's appropriate. This leaves the agent to infer usage based on the tool name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• transcribe_fileB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions the service provider ('AssemblyAI') but does not describe key behaviors such as authentication needs, rate limits, processing time, error handling, or what the transcription output looks like (e.g., text format, confidence scores). This leaves significant gaps for an agent to understand how to use the tool effectively.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose ('Transcribe audio from a local file path') and includes essential context ('using AssemblyAI'). There is no wasted wording, and it is appropriately sized for the tool's complexity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (2 parameters, nested objects, no output schema, and no annotations), the description is incomplete. It lacks information on behavioral traits (e.g., authentication, processing), output format, and detailed usage guidelines. Without annotations or an output schema, the description should compensate more to help an agent invoke the tool correctly, but it does not.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters ('filePath' and 'options') with descriptions. The description adds no additional parameter semantics beyond what the schema provides, such as explaining the 'options' sub-parameters (e.g., what 'speaker_labels' does) or file format requirements. Baseline 3 is appropriate when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Transcribe audio') and resource ('from a local file path'), distinguishing it from sibling tools like 'transcribe_url' which handles URLs instead of local files. It also specifies the service provider ('using AssemblyAI'), making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context by specifying 'local file path', suggesting this tool is for local files versus alternatives like 'transcribe_url'. However, it does not explicitly state when to use this versus siblings like 'submit_transcription' or 'get_transcript', leaving some ambiguity about the workflow or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• transcribe_urlB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions the service provider (AssemblyAI) but does not describe key behaviors such as rate limits, authentication requirements, processing time, error handling, or output format. This is inadequate for a tool that performs external API calls.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core functionality ('Transcribe audio from a remote URL') and adds useful context ('using AssemblyAI'). There is no wasted verbiage, making it highly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of an external transcription service with no annotations and no output schema, the description is insufficient. It lacks details on behavioral traits (e.g., rate limits, auth), output format, error conditions, and differentiation from sibling tools, leaving significant gaps for an AI agent to use it effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters ('audioUrl' and 'options') thoroughly. The description does not add any meaningful semantics beyond what the schema provides, such as explaining the purpose of optional settings or providing examples. Baseline 3 is appropriate when schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Transcribe audio') and resource ('from a remote URL'), and distinguishes from siblings like 'transcribe_file' by specifying the URL source. It also identifies the service provider ('using AssemblyAI'), making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for audio transcription from URLs, but does not explicitly state when to use this tool versus alternatives like 'transcribe_file' or 'submit_transcription'. No exclusions or prerequisites are mentioned, leaving usage context somewhat vague.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• submit_transcriptionA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It mentions 'without waiting for completion', which hints at asynchronous behavior, but fails to disclose critical details like how results are retrieved (e.g., via polling or callback), authentication requirements, rate limits, error handling, or what happens on submission failure. For a tool with no annotation coverage, this leaves significant behavioral gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose ('submit audio for transcription') and adds key behavioral context ('without waiting for completion'). There is no wasted verbiage, and every word earns its place by clarifying the tool's asynchronous nature.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of an asynchronous submission tool with no annotations and no output schema, the description is incomplete. It lacks details on how to handle the submission result (e.g., returns a job ID or status), error scenarios, or integration with sibling tools like 'get_transcript'. For a tool that likely involves background processing, more context is needed to use it effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters ('audio' and 'options') thoroughly. The description adds no additional parameter semantics beyond what's in the schema (e.g., it doesn't explain format constraints for 'audio' or default values for 'options'). With high schema coverage, the baseline is 3, as the description doesn't compensate but also doesn't detract.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('submit audio for transcription') and distinguishes it from siblings by specifying 'without waiting for completion', which implies an asynchronous operation. This differentiates it from tools like 'get_transcript' (which likely retrieves results) and 'transcribe_file/url' (which might be synchronous).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context by indicating this is for submitting audio without waiting, which implies it's appropriate for asynchronous processing. However, it doesn't explicitly state when NOT to use this tool or name specific alternatives among the siblings (e.g., 'use transcribe_file if you need immediate results'), leaving some guidance implicit rather than explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.