Server Quality Checklist

Tool Scores

• job_statusA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses the dual behavior (by id or all) and that it is read-only (via '[read]'). However, it does not explain what happens when job_id is invalid, whether results are paginated, or what state jobs can be in. Adequate but not thorough.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, with no wasted words: it states the operation type, resource, and conditional behavior concisely. Perfectly front-loaded and efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has one optional parameter, no output schema, no annotations, and low sibling relatedness, the description covers the core mechanism. But for a listing tool, it could mention what fields each job status entry contains, whether results are sorted, or any rate limits. It is minimally complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so description must compensate. The description explains the 'job_id' parameter's role: if provided, returns status for that job; if empty/omitted, returns all jobs. This adds significant meaning beyond the bare schema—specifically the default behavior. However, it doesn't specify format constraints or validation rules for the id string.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description starts with '[read]' which clearly signals a read operation, and then states the tool checks status by id or lists all jobs. This distinguishes it from siblings like 'download' or 'transcribe' which are clearly different operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use when needing job status or listing current jobs, but does not explicitly say when to use alternatives or when not to use it. No exclusions or sibling comparisons are provided, leaving the agent to infer context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• transcribeA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description bears full responsibility for disclosing behavior. It notes '[writes]' (suggesting mutation), requires a Whisper model, and mentions variable runtime. However, it fails to clarify side effects (e.g., file creation?), error handling, idempotency, or safety aspects. This leaves significant gaps in behavioral understanding for an agent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences long, front-loading the purpose first, then parameter clarification, then prerequisite and performance note. Every sentence contributes distinct value with no redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the sibling tools (list_files, download, get_transcript), the description fits well into a transcription workflow. It explains the prerequisite (Whisper model), the path convention, and the time cost. Minor omission: it could explicitly state that the file must already be downloaded via the download tool, but the phrase 'downloaded file' and the path example imply this. The output schema exists, so return value documentation is not needed. Overall very helpful for an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, requiring the description to explain all parameters. It adequately explains the 'name' parameter (downloads-relative path with example), but entirely omits the 'language' parameter (schematype: string, default: ''). With only one of two parameters described, the description adds only partial value beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action: 'Transcribe a downloaded file locally with Whisper and return the text.' It specifies the verb (transcribe), resource (downloaded file), method (Whisper locally), and output (text). This distinguishes it from sibling tools like get_transcript (which likely retrieves existing transcripts) or list_files/download.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides context: it requires a Whisper model set in settings and warns that it can take a while. This gives implicit usage guidance, but it does not explicitly state when to use this tool over alternatives (e.g., get_transcript) or list conditions where it should not be used. The guidance is adequate but not comprehensive.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• resolveA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description uses '[read]' to indicate a read-only operation, and explicitly states 'NOT downloading' and 'analyze a link... WITHOUT downloading.' This provides behavioral clarity about the tool being non-destructive. Since no annotations are provided, the description carries the full burden; it does so well but could add details like whether it requires authentication or has rate limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, zero filler. The first sentence defines purpose and behavior; the second provides usage guidance. Every word earns its place. No redundancy or unnecessary details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (1 required param, no enums, output schema present), the description covers the core purpose, read-only behavior, and relationship to siblings. It lacks mention of what the output schema contains, but since an output schema exists, the agent can infer return structure. Could justify a 5 as complete for a simple tool, but 4 acknowledges minor gaps like authentication preconditions.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description says 'analyze a link' which implicitly tells the agent that the single parameter 'url' is a link to analyze. This adds semantic meaning beyond the bare schema, which only defines 'url' as a string. With schema description coverage at 0%, the description compensates well, though it could be more explicit about the expected URL format (e.g., HTTP/HTTPS, supported domains).

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool analyzes a link and lists available media items and qualities without downloading. The verb 'analyze' and noun 'link' are specific, and the description differentiates it from 'download' by explicitly noting it does NOT download. However, it could be more precise about what 'media items' means (e.g., video/audio tracks, formats).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use this tool: 'preview what a URL contains before calling download.' It provides a clear exclusion: 'WITHOUT downloading.' The last sentence directly advises the agent to use this tool before the 'download' sibling, giving strong usage guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• downloadA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses that the tool writes to 'vdl' (a local directory), that wait=True blocks and returns a file name, and wait=False returns a job_id. It does not mention potential overwrites, rate limits, or auth requirements, but the core behavior is well covered.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded: a single sentence defining purpose, followed by bullet points for details. Every sentence adds value, and the structure is easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 5 parameters and an output schema, the description covers purpose, usage, and most parameters. It interacts well with sibling tools. The only oversight is the timeout_seconds parameter, but overall it is sufficiently complete for an agent to select and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description must compensate. It explains url (implicit), wait, audio, and section with examples. The only missing parameter is timeout_seconds, which is not addressed. This is a minor gap given the overall clarity.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Download a video or audio from a URL into vdl (best quality).' It uses a specific verb and resource, and distinguishes itself from sibling tools like transcribe and get_transcript by mentioning their interaction.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use wait=True vs wait=False, and how to combine with transcribe/get_transcript. It also explains audio and section options. However, it does not explicitly state when not to use this tool or mention alternatives like job_status for tracking wait=False jobs.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_transcriptA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It clearly states the tool is a read operation ('[read]') and describes the conditional behavior (returns text or instructs to transcribe). However, it does not detail the return format or what happens if the file hasn't been downloaded, missing some transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise, using two short sentences to convey the purpose, condition, and fallback instruction. No words are wasted, and the structure front-loads the key action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has only one parameter, no annotations, but does have an output schema, the description is nearly complete. It covers the core behavior and prerequisite. It lacks some context about error states or file existence requirements, but for a simple tool, this is sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaning to the single required parameter 'name' by implying it must refer to a downloaded file that has been transcribed. Although schema coverage is 0%, the description compensates by providing context for the parameter's expected value. Without additional detail like format or examples, it cannot score a 5.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns transcript text from a downloaded file, using a specific verb ('Return') and resource ('transcript text (.txt)'). It also distinguishes itself by mentioning the prerequisite condition, which differentiates it from sibling tools like 'download' and 'transcribe'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly tells the user when to use the tool ('if it has been transcribed') and when not to use it ('If none exists yet, tells you to run transcribe(name) first'). This provides clear context and an alternative action.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_filesA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and does well. It explicitly marks the tool as a read operation with '[read]', lists return fields (name, size, duration, labels, in_feed), and mentions all filters. This fully informs the agent of the tool's behavior and output.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, front-loaded with the read indicator and core action. Every piece of information earns its place: operation type, resource, filters with examples, and return fields. No filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the output schema exists, the description does not need to detail return values further. It covers filters, operation type, and a general resource overview. I deduct 1 point because the 'limit' parameter is not explained, though its default is visible in the schema; with 0% schema coverage, a brief mention would make it fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It does so by explaining that 'query' is a substring for name/label, 'service' is exemplified with 'youtube', and 'label' is a filter. 'limit' is not mentioned explicitly but has a default visible in schema, and its purpose is inferable. The description adds substantial meaning beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'List downloaded files in vdl' with a specific verb and resource. It also lists optional filters (query, service, label) and return fields, distinguishing it from sibling tools like get_transcript or download.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description says 'Optional filters' and lists them, giving clear context for when to use each filter. However, it does not explicitly mention when not to use this tool versus alternatives, though the read nature and filters are clear enough for an agent.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.