Server Quality Checklist

Tool Scores

• get_jobA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description carries full burden. It clearly discloses the two terminal states (complete with output object, failed with error object), indicates that polling is required, and notes the interval advice. It doesn't mention non-mutating behavior explicitly, but 'fetching state' implies a read-only operation. At best, it could enumerate all statuses (e.g., pending/running) but that's a minor gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the action, then a compact description of return states and polling cadence. Every word earns its place; no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has only 1 parameter, no output schema, and no annotations. The description fully covers what the tool does, how to call it, what to expect in the response, and how often to poll. Nothing critical is missing for an agent to invoke it successfully.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% — the single id parameter is fully described as 'The job id returned by submit_job.' The description adds minimal extra meaning with 'previously submitted' but doesn't need to. Baseline 3 is appropriate given high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description opens with 'Fetch the current state of a previously submitted job' — a specific verb+resource pairing that clearly distinguishes it from siblings (submit_job creates a job, list_jobs lists jobs). It also narrows the scope to previously submitted jobs via the id parameter.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit polling guidance: 'Call this repeatedly (every 30-60s is plenty — jobs take minutes, not seconds) until status is one of complete/failed.' It implies when to use this tool (to check a specific job's status) rather than list_jobs, though it doesn't explicitly name alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_jobsA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses ordering ('newest first'), pagination ('cursor-paginated via starting_after'), and authentication context ('authenticated organisation'). It doesn't describe return format or error behavior, but for a simple list operation these are not critical gaps, and the disclosed traits are genuinely useful.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, each earning its place: the first states the core function and ordering, the second explains pagination and the intended use case. No redundant details, front-loaded with the primary purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the low complexity (two optional params, no output schema) and the presence of sibling tools that clarify the API landscape, the description is adequate. It could have explicitly mentioned the response shape, but the tool's name and purpose imply a list of job objects. The description covers the key contexts: auth scope, ordering, pagination, and when to use it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with descriptions for both parameters: 'limit' and 'starting_after'. The description adds context about pagination via starting_after and ordering, but this mostly reinforces schema info (e.g., 'Cursor: pass the last job id...'). It does not add meaning beyond what the schema already provides, so the baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb+resource pattern: 'List recent jobs' with a clear scope ('for the authenticated organisation') and ordering ('newest first'). It distinguishes from sibling tools by stating 'when you don't have the id to hand,' which contrasts with get_job and implicitly with submit_job.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states the use case: 'Useful for inspecting recent submissions when you don't have the id to hand.' This clearly tells the agent when to use this tool instead of a direct ID-based lookup, and implies it's the correct choice for browsing recent items without a known ID.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• submit_jobA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully discloses the async behavior (immediate 'queued' status), expected wall-clock times for different source types, and special sandbox behavior (canned fixture output). This is transparent about what the caller should expect. No contradictions with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is four sentences, each delivering actionable information: behavior, timing, beta status, and sandbox testing. There is no fluff; every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is comprehensive for a submission tool: it explains the immediate response ('queued' with a job id), how to track progress (get_job), expected performance, beta limitations, and testing options. Although there is no output schema, the description covers the key return behavior. The nesting and options are well documented in the schema, and the description adds the behavioral context necessary for correct usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides full descriptions for all parameters (100% coverage), so the baseline is 3. The description adds beyond that by giving timing estimates that depend on source type and include_trailer, enriching the semantics of those parameters. It does not explain each parameter individually, but the schema does, and the added context about time is useful.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a clear verb and resource ('Submit a podcast for processing'), and distinguishes itself from siblings by noting that it returns a job id immediately and that polling is done via get_job. This makes its role unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides explicit guidance: the tool submits a job asynchronously, and callers should use get_job to poll status. It also clarifies constraints for youtube_url (beta, rejected for live keys) and suggests sandbox keys for deterministic testing, which are valuable usage conditions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.