Server Quality Checklist

Tool Scores

• extend_videoC
  2.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden for behavioral disclosure. It states the tool returns a task id, status, and output URLs, which hints at asynchronous behavior, but it does not disclose that the task may initially be pending, that wait/polling parameters exist, or that this is a paid/compute-consuming operation. Minimal behavioral context is provided.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no wasted words. It efficiently conveys the primary action and return behavior. Excellent conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having 9 parameters, 3 required, no output schema, and no annotations, the description provides only the basic purpose and return format. It lacks usage guidance, parameter explanations, and asynchronous behavior details. This is inadequate for a tool of this complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is only 22% (only wait and model have descriptions). The tool description adds no explanation for required parameters such as source_task_id, prompt, or output_resolution. It fails to compensate for the low schema coverage, leaving the agent without parameter meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it creates a Runway task on RunAPI specifically for extending a video. The parenthetical '(extend video)' distinguishes it from the sibling text_to_video tool. However, it does not explicitly name the alternative or contrast with text_to_video, so it falls short of a perfect 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like text_to_video. It does not mention any preconditions, exclusions, or specific scenarios. This is a clear gap, especially given the sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_pricingA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations present, the description carries full burden but only states it's a lookup operation. It omits any behavioral traits like authentication needs, rate limits, or output format.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with no unnecessary words. It is appropriately concise, though could incorporate more context without sacrificing brevity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple lookup tool with two optional parameters and no output schema, the description sufficiently explains the purpose. It could note default behaviors, but the schema handles that.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the schema fully documents both parameters. The description adds no extra meaning beyond what the schema's enum descriptions already provide.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('look up RunAPI pricing') and specific resource ('runway model line'), distinguishing it from sibling tools which deal with video generation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for pricing lookups but provides no explicit guidance on when to use it over siblings or any exclusions/alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• text_to_videoA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses the return value (task id, status, output URLs) and mentions RunAPI, but it does not explain the asynchronous nature, potential costs, or any side effects. With no annotations, the description carries the full burden and only partially fulfills it.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that front-loads the primary action and return info. It contains no filler or redundant details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For an 11-parameter tool with no output schema and no annotations, this description is far from complete. It leaves critical gaps around async processing, parameter usage, and interpretation of the task status, making it inadequate for effective tool selection and invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is only 18%, so the description needed to compensate by explaining parameters. It does not, though it gives a general hint that 'prompt' is the text input. Optional parameters like wait, timeout_ms, and callback_url remain unexplained.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Create') and the resource ('Runway task'), with the parenthetical 'text to video' specifying the modality. It distinguishes from siblings like extend_video and get_task by focusing on creation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use for generating a new video from text, but provides no explicit guidance on when to use it vs. alternatives like extend_video or get_task. It does not mention exclusions or the optional waiting behavior.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_taskA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It clearly indicates a read-only operation ('Fetch'), which implies non-mutating behavior. However, it does not disclose error handling, rate limits, or any side effects, so transparency is minimal but adequate for a simple fetch.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with zero redundancy. It concisely states the action and target resource without unnecessary detail, earning a perfect score for conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Since there is no output schema, the description should clarify what the tool returns. It mentions 'latest result payload' but is vague about the structure or additional fields. For a simple read tool with well-documented parameters, this is adequate but not comprehensive.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema provides 100% coverage for both parameters (task_id and action) with clear descriptions. The tool description does not add any further parameter-level meaning beyond what the schema already provides, so the baseline score of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description starts with the specific verb 'Fetch' and identifies the resource as 'current status and latest result payload for a runway task.' This clearly distinguishes it from sibling tools like extend_video and text_to_video, which create tasks, making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for checking task status after creation via context from sibling tools, but it does not explicitly state when to use this tool versus alternatives, nor does it mention exclusions or prerequisites. The guidance is implied rather than direct.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• loginA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full behavioral burden. It discloses key behaviors: opens a browser, uses PKCE, saves to a config file. However, it omits details like error handling or whether it checks existing credentials.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with no unnecessary words. It is concise and front-loaded with the action and resource.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool is a simple login with one optional parameter and no output schema, the description covers the essential action and side effect. It could mention return value or behavior on already authenticated state, but overall it is sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% for the single parameter 'force', and the schema already describes its meaning. The description adds no further parameter context, so baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Authenticate RunAPI by opening a browser PKCE login flow and saving the API key' which is a specific verb and resource. It distinguishes the login tool from unrelated sibling tools like check_pricing or extend_video.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage (need to authenticate) but does not explicitly state when to use it versus alternatives, nor does it mention prerequisites or exceptions like already being logged in. The 'force' parameter is described in the schema but not in the description.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.