Server Quality Checklist

Tool Scores

• get_transcript_statusA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses that the task is async, returns status/progress/result, and advises polling until terminal states. This is sufficient for a simple status-checking tool, though it doesn't mention potential errors or rate limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is highly concise and front-loaded, with two sentences covering purpose, return value, and usage pattern. Every sentence earns its place with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter tool with no output schema, the description adequately explains purpose, return type, and when to call. It's complete enough for an agent to invoke correctly, though it doesn't detail the response structure beyond high-level fields.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already provides a clear description for task_id, and the description reinforces that it comes from transcribe_url or transcribe_file. With 100% schema coverage, the description doesn't add new parameter semantics, meeting the baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool checks the status of an async transcription task. This distinguishes it from sibling tools that initiate transcription (transcribe_url/transcribe_file) or list transcripts (list_transcripts).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly instructs to poll periodically until status is 'completed' or 'failed', providing clear usage context. However, it doesn't explicitly say when not to use it versus listing or initiating, though the sibling names make those exclusions obvious.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• transcribe_fileA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the behavioral disclosure burden. It discloses supported formats, Whisper as the underlying model, and the behavior for long files (async, segment splitting). It does not mention output structure or side effects, but the key operational traits are covered and there is no contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, front-loaded with the core purpose, followed by format support and targeted usage advice. Every sentence contributes directly to effective tool usage, with no redundant or repetitive content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 5 parameters, no output schema, and no annotations, so the description should clarify return values or outcomes. It explains local-file handling and long-file guidance well, but it does not state what the tool returns (e.g., transcript text, task_id for async), which is a gap given the absence of an output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, providing baseline 3. The description adds value by explaining when to use async_mode and how to adjust segment_minutes for 1h+ files, supplementing the schema's parameter descriptions beyond just listing defaults.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool transcribes local audio/video files using Whisper, with a specific verb (transcribe) and resource (local file). It also lists supported formats and distinguishes itself from sibling transcribe_url by emphasizing 'local' file input.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear context for when to use this tool (local files, with format support) and provides concrete guidance for long files (use async_mode, adjust segment_minutes). However, it does not explicitly state when not to use it or mention the sibling transcribe_url for remote files, so it lacks explicit alternative naming.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• transcribe_urlA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses the three-tier strategy (subtitles, Whisper, closed-platform guidance) and sync/async behavior. It doesn't cover error handling or permissions, but the tier strategy provides substantial behavioral insight.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the main purpose, structured into clear tiers, and every sentence contributes actionable information without redundancy. It is concise yet comprehensive.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 7 parameters, no output schema, and no annotations, the description covers the workflow well: tiers, sync/async, polling, and closed-platform guidance. It could explicitly state the sync return value, but the tier strategy and usage notes make it largely complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds value by explaining async_mode for long videos, segment_minutes for 1h+, and sync mode default, enriching the schema's parameter descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: 'Transcribe a video from URL using three-tier strategy.' It clearly distinguishes from siblings by mentioning polling with get_transcript_status and contrasting with transcribe_file (via the URL versus file input).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit guidance is given: 'For videos under 30 minutes, use sync mode (default)... For longer videos, set async_mode=true... poll with get_transcript_status.' It also advises adjusting segment_minutes for 1h+ videos, clearly indicating when to use which mode.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_transcriptsA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. Although it does not explicitly say "read-only," the verbs "List" and "Returns" strongly imply a non-mutating operation, and the choice of "completed" adds a behavioral constraint. It does not disclose details like pagination or authentication, but for a simple listing tool the behavior is adequately transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences, front-loaded with the core purpose. The first sentence states the action, the second covers return type and a pointer to a sibling tool. Every sentence earns its place with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple (no parameters, no annotations), but an output schema is present, so the description doesn't need to detail return values. The description covers the contents (list of completed tasks with metadata) and references the next logical step (getting full text via get_transcript_status). This is complete for the tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and the schema is empty. The description adds no parameter-specific details because there are none to document. Per the calibration rubric, a baseline of 4 applies for 0-parameter tools, and the description does not detract from that.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb+resource phrase "List all completed transcripts," which clearly states the tool's purpose. It distinguishes itself from sibling tools by noting that get_transcript_status is used for full transcript text, whereas this tool returns a list with metadata.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly instructs the agent to use get_transcript_status with a task_id when full transcript text is needed, providing a clear alternative and when-to-use distinction. This directly addresses when to use this tool versus its siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.