Server Quality Checklist

Tool Scores

• check_pricingC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full responsibility for behavioral disclosure. It only states 'look up pricing' without detailing side effects, authentication requirements, rate limits, or any output format. For a tool with zero annotation coverage, this is insufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with no wasted words. It is appropriately sized for a simple lookup tool, though it could benefit from additional context without becoming verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has two optional enum parameters, no output schema, and no annotations. The description covers the basic purpose but omits details like what the 'action' parameter does or what the pricing response contains. Given the simplicity, it is minimally viable but leaves gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, with both parameters fully described in the input schema (enums and descriptions). The description adds minimal value beyond stating the model line, but given high schema coverage, a baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool looks up pricing for a specific model line ('runway-aleph'). The verb 'look up' implies a read operation, and the resource is well-defined. It distinguishes itself from siblings like 'edit_video' and 'get_task' by the specific function of pricing lookup.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is given on when to use this tool versus alternatives. While siblings are clearly different tools, the description does not specify context, prerequisites, or exclusion criteria. Agents have no information on appropriate usage scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• edit_videoC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must disclose behavioral traits but only states the return format (task id, status, output URLs). It omits important details such as the operation being asynchronous (evident from wait/poll parameters), potential side effects (initiating a processing job), or any auth/cost implications. This is insufficient for a task-creating tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the primary action and includes the key output behavior. No redundant words or unnecessary detail; it is appropriately sized for the information it conveys.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For an 11-parameter tool with no annotations and no output schema, the description is too sparse. It does not explain the editing workflow, parameter purposes, or how it fits with sibling tools. The return value mention is helpful but insufficient for a complex tool requiring prompt-based video edits with multiple optional controls.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is only 18%, and the description provides no additional parameter explanations. The required 'prompt' and 'source_video_url' are not elaborated, nor are optional parameters like aspect_ratio, reference_image_url, or callback_url. The description fails to compensate for the sparse schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states the action ('Create a Runway Aleph task') and resource ('on RunAPI'), with the parenthetical 'edit video' clarifying the domain. It clearly distinguishes from siblings like login, get_task, and check_pricing, which serve different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. It does not mention that get_task should be used to check status, nor does it specify prerequisites or scenarios for which this tool is appropriate. The usage is only implied by the tool's name and the mention of 'edit video.'

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_taskA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. The verb 'Fetch' implies a read-only operation, and it specifies what is returned (status and result payload). However, it does not disclose error behaviors, authentication needs, or whether results can be polled repeatedly. This is adequate but lacks richer safety or side-effect context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no filler. It efficiently communicates the core function and includes the key resource ('runway-aleph task') in a compact way.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a two-parameter tool with no output schema and no annotations, the description gives a basic understanding of the return type (status and payload) but omits details about polling behavior, status interpretation, or connection to sibling tools like edit_video. It is minimally complete but leaves room for ambiguity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides 100% description coverage for both parameters (action and task_id), each with clear schema descriptions. The tool description adds no additional parameter semantics beyond what the schema covers, so the baseline score of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Fetch') and the specific resource ('current status and latest result payload for a runway-aleph task'). This distinguishes it from sibling tools like login, edit_video, and check_pricing, making the purpose immediately clear.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for checking on an asynchronous task (likely after using edit_video), but it does not explicitly state when to use this tool versus alternatives or include any exclusionary guidance. No alternatives are named, and the when-to-use context is only inferred.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• loginA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It transparently describes the login method, the side effect of saving the API key to a specific file path, and implies user interaction via the browser flow. It does not cover failure modes or existing credentials, but the core behavior is clearly disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that leads with the purpose, then details the method and location. Every word earns its place, and there is no redundant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one optional parameter, no output schema), the description sufficiently covers the essential behavior. It explains what the tool does and where it persists data. It omits details like return values or error cases, but these are not critical for tool selection and invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema covers 100% of the parameter's description (the 'force' boolean's behavior is fully explained in the schema). The tool description adds no additional parameter context, so the baseline score of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('Authenticate'), resource ('RunAPI'), and method (browser PKCE login flow). It also specifies the outcome (saving the API key to a config path), distinguishing it from unrelated sibling tools like edit_video and check_pricing.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The usage is implied: the tool is for authenticating RunAPI. However, the description does not explicitly state when to use it versus alternatives, nor does it mention conditions under which it should not be used (e.g., already having valid credentials). This is adequate but not explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.