Server Quality Checklist

Tool Scores

• get_transcriptC
  2.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions 'GET /api/v1/transcripts/{video_id}', implying a read-only HTTP operation, but doesn't specify authentication needs, rate limits, error handling, or what 'full transcript' entails (e.g., format, length limits). This leaves significant gaps for a tool with 5 parameters and no output schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is brief and front-loaded with the core purpose ('Get full transcript for a video'), followed by an API endpoint detail. It avoids unnecessary elaboration, though the endpoint detail might be redundant if the agent already knows the tool's name and context. Overall, it's efficient with minimal waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (5 parameters, 20% schema coverage, no annotations, no output schema), the description is inadequate. It doesn't explain return values, error conditions, or behavioral nuances like how parameters interact (e.g., 'id' vs. 'source'). For a tool that retrieves data with multiple filtering options, more context is needed to ensure correct usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is low at 20%, with only one parameter ('id') having a description. The tool description adds minimal value beyond the schema, mentioning 'video_id' in the path but not explaining other parameters like 'language', 'source', or 'include_timestamps'. It fails to compensate for the poor schema coverage, leaving most parameters semantically unclear.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Get full transcript') and resource ('for a video'), which provides a specific verb+resource combination. However, it doesn't differentiate from sibling tools like 'list_transcripts' or 'transcribe_v2', which might offer overlapping functionality for transcript retrieval or generation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'list_transcripts' (which might list multiple transcripts) or 'transcribe_v2' (which might generate new transcripts). It only states what the tool does without context about prerequisites, exclusions, or comparative use cases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• delete_transcriptC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states it's a deletion operation, implying it's destructive, but lacks details on permissions, reversibility, rate limits, or response behavior. The mention of 'bulk-delete' hints at batch processing, but no further context is given. This is inadequate for a mutation tool with zero annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded and concise, using a single sentence that efficiently combines the API endpoint, action, and parameters. Every word earns its place, with no redundant information, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given this is a destructive mutation tool with no annotations and no output schema, the description is incomplete. It lacks critical context such as error handling, confirmation requirements, or what happens post-deletion. Siblings like 'get_transcript' suggest read operations, but no integration guidance is provided. More detail is needed for safe and effective use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with clear descriptions for 'ids' and 'video_id'. The description adds minimal value by mentioning deletion 'by ids or by video_id', which aligns with the schema but doesn't provide additional semantics like format examples or usage trade-offs. Baseline 3 is appropriate as the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Delete transcripts') and the resource ('transcripts'), and specifies the method ('POST /api/v1/transcripts/bulk-delete') and deletion criteria ('by ids or by video_id'). It distinguishes from siblings like 'get_transcript' or 'list_transcripts' by being a deletion operation, though it doesn't explicitly name alternatives. The purpose is specific but could be more explicit about sibling differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It mentions deletion criteria but does not specify scenarios, prerequisites, or exclusions (e.g., when to use 'ids' vs 'video_id', or if there are restrictions). With siblings like 'get_transcript' and 'list_transcripts', there is no explicit comparison or usage context provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_statsB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It states the tool retrieves stats, implying a read-only operation, but doesn't disclose behavioral traits such as authentication needs, rate limits (beyond mentioning it as a stat), error handling, or response format. This leaves significant gaps for a tool with no annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that lists all key stats without unnecessary words. It is front-loaded with the verb 'Get' and directly enumerates the resources, making it easy to parse and understand quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 0 parameters, 100% schema coverage, and no output schema, the description is minimally adequate. It states what stats are retrieved but lacks details on response format, error cases, or usage context. For a simple stats retrieval tool, this is acceptable but leaves room for improvement in behavioral transparency.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters, and the input schema has 100% coverage (empty object). The description doesn't need to add parameter details, so it appropriately focuses on the tool's purpose. A baseline score of 4 is given as it compensates for the lack of parameters by clearly stating what stats are retrieved.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: retrieving statistics including credits left, transcripts created, plan, and rate limit. It uses a specific verb ('Get') and lists the resources, though it doesn't explicitly differentiate from sibling tools like 'get_transcript' or 'list_transcripts' beyond the type of data returned.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. The description lists what stats are retrieved but doesn't mention context, prerequisites, or comparisons to sibling tools like 'list_transcripts' for transcript-related queries or 'get_transcript' for individual transcript details.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_transcriptsB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions 'List or search' but doesn't describe key behaviors such as pagination handling (implied by 'page' parameter), default values, error conditions, or response format. For a read operation with multiple parameters, this leaves significant gaps in understanding how the tool behaves beyond basic functionality.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized with two sentences that are front-loaded: the first states the purpose, and the second provides usage guidance. There's no wasted text, and it efficiently conveys key information. However, it could be slightly more structured by explicitly separating listing vs. searching, but it's still highly concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (6 parameters, no output schema, no annotations), the description is moderately complete. It covers the basic purpose and search usage but lacks details on behavioral aspects like response format, error handling, or interactions with parameters. Without annotations or output schema, more context is needed for full agent understanding, making it adequate but with clear gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is high at 83%, so the baseline is 3. The description adds minimal value beyond the schema: it reiterates the 'search' parameter's purpose ('find by video id, title, or transcript content'), which is already covered in the schema description. It doesn't explain other parameters like 'status' enum values or 'include_segments', so it doesn't compensate for the 17% coverage gap, resulting in an adequate but not enhanced score.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'List or search user transcripts' with the verb 'list/search' and resource 'user transcripts'. It distinguishes from siblings like 'get_transcript' (singular) and 'delete_transcript' (deletion) by indicating it handles multiple transcripts. However, it doesn't explicitly differentiate from 'search' functionality in other tools, keeping it from a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides some usage context: 'Use search to find by video id, title, or transcript content,' which implies when to use the search parameter. However, it lacks explicit guidance on when to use this tool versus alternatives like 'get_transcript' for a single transcript or how it relates to other siblings. No exclusions or clear alternatives are mentioned, making it only implied usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• transcribe_v2B
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions 'Fast caption-based transcript' which hints at performance but doesn't specify speed, reliability, or error handling. It doesn't disclose authentication needs, rate limits, or what happens if captions are unavailable. The POST method implies a write operation, but there's no clarity on whether this creates new resources or has side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise with just two sentences. The first sentence states the endpoint and core functionality, while the second provides key usage constraints. Every word earns its place with zero redundancy or fluff. It's appropriately front-loaded with the most critical information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (4 parameters including a nested object), no annotations, and no output schema, the description is insufficiently complete. It doesn't explain what the tool returns (transcript format, structure, or potential errors), doesn't cover all behavioral aspects, and leaves gaps in parameter understanding despite the schema doing some work. For a tool that presumably creates new transcripts, more context is needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 75%, so the schema already documents most parameters well. The description adds minimal value beyond the schema: it mentions 'manual or auto captions only' which relates to the 'source' parameter but doesn't explain the 'auto (manual first, fallback to auto)' behavior detailed in the schema. No additional parameter semantics are provided, so it meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Fast caption-based transcript (no ASR)' specifies it generates transcripts from captions rather than automatic speech recognition. It mentions 'manual or auto captions only' which further clarifies the input source. However, it doesn't explicitly differentiate from siblings like 'get_transcript' or 'list_transcripts' beyond the 'POST /api/v2/transcribe' endpoint reference.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides some usage context: 'Use manual or auto captions only' and 'no ASR' implies when this tool is appropriate versus alternatives that might use ASR. However, it doesn't explicitly state when to use this versus siblings like 'get_transcript' (which presumably retrieves existing transcripts) or 'delete_transcript'. The guidance is implied rather than explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.