Server Quality Checklist

Tool Scores

• summarize_transcriptC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It fails to mention that this tool invokes an LLM via the gateway, depends on a configurable model, or sends transcript content to an external model — all important behavioral traits for an AI agent deciding whether the operation is appropriate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with no wasted words, which is concise in the literal sense. However, it is under-sized for a tool that has four parameters and no output schema, so it does not fully earn its place as the sole behavioral guide.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema and no annotations, the description should explain what the summary looks like, how multiple transcripts are combined, and what role context and answer_format play. It does none of this, leaving an agent to infer key execution details.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides descriptions for all four parameters, so the description does not need to restate them. The phrase 'one or more transcripts' does reinforce transcript_ids having minItems=1, but it adds no deeper meaning about how model, context, or answer_format affect the result.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Summarise') and a clear resource ('one or more transcripts'), making the core function immediately obvious. It does not, however, differentiate itself from the sibling ask_transcript, which could also be used to derive meaning from transcripts.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no guidance on when to use summarize_transcript versus ask_transcript or get_transcript. The description only states what it does, not the conditions that make it the right choice.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ask_transcriptC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations are absent, so the description carries full responsibility for behavioral disclosure. It only says the tool answers from transcripts; it does not disclose that an LLM is involved, potential hallucination or grounding limitations, output format, or whether the operation is read-only. Some behavior is implied, but little is actually transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single short sentence that front-loads the tool's purpose. It contains no wasted words, though the phrasing 'Ask a question answered from' is slightly awkward and the conciseness comes at the expense of useful behavioral and usage context.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the small parameter surface and fully documented schema, an agent can infer the basic call pattern: provide transcript IDs and a question. However, there is no output schema, no note about return value, no guidance versus sibling tools, and no limitation disclosure, leaving meaningful gaps for an agent that needs to actually invoke this correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  All three parameters (transcript_ids, question, model) are fully described in the input schema, including the model default. The description adds no parameter-level meaning beyond what the schema already provides, so the baseline score of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a clear action ('Ask a question') and resource ('transcripts'), and the phrase 'answered from one or more transcripts' conveys an understanding/Q&A role rather than a retrieval/list role. It does not explicitly name a sibling like search_transcript or summarize_transcript to differentiate itself, so it stops just short of a 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no explicit guidance on when to use this tool versus alternatives such as search_transcript, summarize_transcript, or get_transcript. The description only implies the obvious case—asking a natural-language question over transcripts—but provides no exclusions, preferences, or context for choosing it over sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_speaker_turnsB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full transparency burden. It discloses the speaker_labels prerequisite and the general return concept, but says nothing about ordering, pagination, errors for unlabeled transcripts, or how max_turns affects results. This is a significant gap for a data-retrieval tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single sentence that front-loads the core purpose with no filler. It is appropriately succinct, though slightly too terse to carry all necessary behavioral context on its own.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no annotations, no output schema, and siblings like get_transcript and search_transcript, this description is incomplete. It covers the speaker_labels prerequisite but not the return shape, failure behavior, or relationship to sibling tools, leaving an agent to guess important call-time details.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so both parameters are already documented. The description adds no new parameter-level meaning beyond restating the speaker_labels condition already present in the transcript_id schema, so the baseline 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the action ('Get') and the resource ('who said what' / speaker turns), and adds the relevant condition of a transcript made with speaker_labels. It does not explicitly distinguish itself from get_transcript or search_transcript, so it stops short of a 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It communicates when the tool is appropriate: for transcripts made with speaker_labels. However, it provides no guidance on when not to use it or which sibling alternative to choose, such as get_transcript for transcripts without speaker labels.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_transcriptsB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must carry the burden. 'List recent transcripts' conveys a read-only, account-scoped, recency-ordered operation, but it does not explicitly state non-mutation, pagination behavior, or what fields the returned transcripts contain. This is minimal but not misleading.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is one short, front-loaded sentence with no filler. The only slight weakness is the undefined term 'recent,' but structurally it is appropriately concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple and has full parameter documentation, so the description is minimally adequate for invocation. However, with no output schema and no annotations, the return format/fields and precise ordering semantics are left unspecified.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema fully documents both parameters, including default and maximum for limit and the status enum, so the baseline 3 applies. The description adds no parameter-level meaning beyond what the schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific operation ('List') and resource ('transcripts'), and narrows scope to 'recent' and 'on the account.' It is clear at a glance and distinguishable from the singular get_transcript or search_transcript, though it does not explicitly name those siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use this tool: when the agent needs a list of recent account transcripts. It does not mention alternatives such as search_transcript or get_transcript, nor does it state when not to use it, so the usage guidance is only implicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• transcribeB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the disclosure burden. It does reveal a key behavioral trait: it waits for the result by default, which implies async behavior can be opted out of. But it does not explain what happens when wait is false, what the eventual response looks like, or whether a stored transcript is created for later retrieval.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two short sentences with no wasted words. The core operation is front-loaded, and the key default behavior is stated immediately after. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  There is no output schema and no annotation coverage, yet the tool has six parameters and a potentially long-running async flow. The description does not explain how to retrieve results, what wait=false returns, or how this interacts with sibling transcript tools. This leaves a meaningful gap for correct usage beyond the default path.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all six parameters. The description adds no parameter-level detail beyond what the schema provides; even the wait default is already stated in the wait parameter's schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: transcribe an audio or video file from a URL. This is clear, but it does not explicitly distinguish the tool from siblings like get_transcript or ask_transcript, so it falls short of full differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The intended use is implied: call this when you need to transcribe a media file from a URL. However, there are no exclusions or alternatives mentioned, and given the large sibling set, the agent gets no explicit guidance about when to prefer this over transcript retrieval or search tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_subtitlesA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the core operation but does not mention error behavior for incomplete transcripts, return format details, permissions, or whether the operation is read-only. For a tool with no annotation coverage, this is a significant gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single, focused sentence that conveys the essential purpose and precondition without redundancy. Every word adds value, and the key constraint ('completed transcript') is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple with only three parameters and no output schema. The description covers the operation and precondition, but without annotations or an output schema, it omits details about the exact return value (e.g., raw subtitle text vs. a download URL) and error cases. This is adequate but not fully complete for an agent to predict tool behavior confidently.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already documents all parameters with descriptions (100% coverage), so the baseline is 3. The description adds little beyond the schema, merely reiterating 'SRT or VTT' and 'completed transcript'. The optional chars_per_caption parameter is not mentioned in the description, but the schema handles it adequately.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Get', the specific resource 'SRT or VTT subtitles', and the condition 'for a completed transcript'. This distinguishes it from sibling tools like get_transcript and get_speaker_turns, making the tool's purpose unmistakable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context by specifying 'for a completed transcript', implying this tool is only appropriate after transcription is done. However, it does not explicitly mention alternatives or when not to use this tool, leaving some inference to the agent.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_transcriptA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must carry behavioral disclosure. The verb 'Fetch' conveys a read-only operation and 'including its status if it is still running' hints at asynchronous/polling behavior. However, it does not describe the return structure, error behavior, or auth expectations, so transparency is only partial.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no wasted words. It states what is fetched, how it is identified, and the status behavior efficiently.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple two-parameter fetch tool, the description plus schema covers the main calling pattern. But with no output schema and no annotations, it leaves ambiguity about the exact response shape and how status values behave when the transcript is no longer running.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the parameters are already well documented in the schema: transcript_id is the id from transcribe, and max_characters trims text with a default of 20000. The description adds no extra parameter meaning, so the baseline of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: 'Fetch a transcript by id.' Adding 'including its status if it is still running' gives a distinctive behavioral detail that separates it from siblings like list_transcripts and get_subtitles.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'by id' implies use when a transcript_id is already available, but there is no explicit guidance on when to prefer this over siblings or when not to use it. The parameter schema's 'Id returned by transcribe' note provides indirect context, but the description itself does not offer alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_transcriptA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the behavioral burden. It doe disclose that the tool returns counts and timestamps, but it does not describe matching semantics, result structure, or any edge cases. This is basic but adequate transparency with clear gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two short sentences with no filler. The verb-resource statement is front-loaded and the output expectation follows directly. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple two-parameter tool with no output schema, the description gives enough to attempt an invocation, but it leaves return-format details and search behavior unspecified. It is complete at a high level, yet lacks precision an agent might want before calling.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the schema already documents both transcript_id and words. The description adds no parameter-level detail beyond what the schema provides, so the baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific verb and resource: 'Find where words are spoken in a transcript.' It also names the output ('counts and timestamps'), which clearly distinguishes it from sibling tools like get_transcript, get_speaker_turns, or ask_transcript.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Clear context for when to use it: whenever an agent needs to locate where particular words occur in a transcript and get counts/timestamps. It does not explicitly list exclusions or alternatives, so it misses the full 5, but the intended use is directly inferable.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• upload_fileA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must carry behavioral disclosure. It states the core action and the output contract (a URL accepted by transcribe), which is useful. Yet it omits details such as whether uploading is synchronous, how long the URL remains valid, any authentication/privacy implications, or how oversized files are refused—though the latter is captured in the schema's max_size_mb field. The description is not misleading, but it is thin on behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  One sentence, direct, front-loaded, and free of filler. Every word contributes: 'Upload' is the action, 'local audio or video file' is the input, and 'URL that transcribe accepts' provides the output and integration context.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no annotations and no output schema, the description is adequate for a simple upload. It explains the essential purpose and the output shape. However, it lacks information on error conditions, the exact response format beyond 'a URL', and any prerequisites or side effects such as file persistence or cleanup. Given the schema covers parameters, this is acceptable but not fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description adds no additional parameter-level meaning beyond noting the file is local and audio/video, which mirrors the schema's file_path description. It does not clarify max_size_mb's default or limits beyond what the schema already states.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the tool's function: uploading a local audio or video file and returning a URL for downstream use. It distinguishes itself from sibling tools by framing the output as the input for transcribe, making its role in the workflow explicit.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'URL that transcribe accepts' establishes when to use this tool: when a user needs to make a local file available for transcription. It implicitly positions this as the upload step before transcribe, and none of the sibling tools overlap with this action. However, it does not explicitly state when not to use it or mention alternative approaches.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.