Server Quality Checklist

Tool Scores

• start_job_fileA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description must carry behavioral disclosure. It reveals async nature and job ID return but lacks detail on failure modes, file read mechanism, or size limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no wasted words. It front-loads the purpose and follows with usage guidance, achieving maximum conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given good schema coverage and sibling context, the description adequately covers purpose and usage. Missing output schema is partially compensated by mentioning job ID and polling pattern. Could hint at return structure.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. Description adds minimal value by emphasizing the absolute path requirement, but does not explain model, language, or service_mode beyond what schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool uploads a local audio/video file, returns a job id immediately, and is async. It distinguishes from siblings like start_job_url (URL input) and transcribe_file (sync).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly advises using this for long recordings or when polling with get_job_status, implying alternatives for other cases. However, it does not name all sibling tools (e.g., transcribe_file) directly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• cancel_jobA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It discloses the key failure condition (job already terminal), which is critical for safety. No hidden behaviors are implied.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, zero waste. The first sentence states the primary action, the second adds a crucial constraint. Perfectly front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter cancellation tool with no output schema, the description covers the essential behavior and a key edge case. Could mention success side effects (e.g., job state change), but not required for minimal completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and the parameter description is adequate. The tool description adds no extra semantic information beyond the schema's 'job_id' description, so baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb-resource combination ('Cancel...transcription job') and clearly distinguishes from siblings like 'get_job_status' (status query) and 'transcribe_file'/'start_job_*' (job creation).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description states when to use (queued/processing jobs) and implicitly when not to (terminal jobs will fail). It could explicitly reference alternatives like 'get_job_status' to check status first, but the context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_job_statusA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Describes basic behavior (fetch status, include transcript when done, poll). No annotations exist, so description carries full burden. It doesn't mention error handling, rate limits, or authentication needs, but the core read-only behavior is clear.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences that front-load the primary function and include essential usage instruction (poll). No unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter, no output schema), the description adequately covers its purpose and usage pattern. Could be slightly more explicit about return value, but polling instruction compensates.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Single parameter job_id has schema description 'Job id returned by a create or wait-first transcription tool.' The tool description adds context that the id comes from specific tools, which is valuable beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states it fetches job status and transcript when completed, with specific verb 'Fetch' and resource 'status of a transcription job'. Distinguishes from sibling tools (transcribe, start, cancel) by focusing on monitoring/polling.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says to poll until completed/failed/cancelled, guiding usage as a polling mechanism. Does not explicitly mention when not to use, but the polling instruction implies it's for ongoing jobs.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• transcribe_urlA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses the synchronous blocking nature, timeout handling, and asynchronous fallback. However, it does not explicitly describe the successful return value (presumably the transcript), which is a minor gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three concise sentences front-loading the core action, usage guidance, and alternative. No extraneous words; every sentence serves a purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  While usage and timeout behavior are well-covered, the description omits the nature of a successful response (e.g., job object vs. transcript text). Without an output schema, this gap reduces completeness for a 5-parameter tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema provides complete descriptions for all 5 parameters (100% coverage). The tool description does not add additional parameter-level meaning beyond reaffirming the URL requirement. Baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('start transcription from a public HTTPS URL') and resource. It distinguishes this tool from sibling start_job_url by specifying use case (short/medium media needing immediate transcript vs. long audio for polling).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to prefer this tool ('short/medium media when you need the transcript in this turn'), timeout behavior ('returns job_id — then call get_job_status'), and when to use alternative ('For long audio you plan to poll yourself, use start_job_url instead').

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• start_job_urlA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description must carry burden. It discloses async behavior, job id return, and polling mechanism. Lacks details on rate limits, authentication, or cancellation, but is sufficient for a job creation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, no wasted words. Front-loaded with purpose and key usage instruction. Highly efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 4 parameters, no output schema, and moderate complexity, description covers main aspects: async flow, polling, and cancellation. Could mention expected job id format, but overall complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. Description adds minimal extra meaning beyond schema for url (emphasizes 'public HTTPS'). Other parameters not elaborated. Adequate but not exceptional.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the verb (create), resource (transcription job from URL), and async nature. Distinguishes from synchronous tools and mentions return of job id.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says 'Use for long media or deferred workflows' and directs to poll with get_job_status. Implicitly contrasts with synchronous alternatives. Provides clear usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• transcribe_fileA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses wait behavior, timeout handling, and path requirement. Could mention response format on success or file size limits, but core behavior is well covered.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three focused sentences: action, usage advice, edge case+alternative. No wasted words, front-loaded with key info.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers purpose, usage, timeout, and alternative. Lacks explicit mention of normal success return (i.e., transcript text) but handles the main behavioral gap (timeout). Satisfactory given schema richness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with clear descriptions for all 5 parameters. The description reinforces the path constraint but adds little beyond schema. Baseline 3 appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's action ('Upload a local audio/video file and wait until transcription finishes') and distinguishes it from siblings like 'transcribe_url' (remote URL) and 'start_job_file' (no wait).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly advises when to use ('Prefer for short/medium clips when you need the transcript in this turn') and when not to, with a direct alternative ('For long audio you plan to poll yourself, use start_job_file instead'). Also explains timeout fallback.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.