Server Quality Checklist

Tool Scores

• modify_videoB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It only states that a task is created and returns an ID/status/output URLs, but fails to mention critical behavior such as asynchronous nature, the optional 'wait' parameter, polling, or potential side effects. This is insufficient for a task-creation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that front-loads the primary purpose and includes the return value. There is no fluff or redundant wording; every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 8 parameters and no output schema or annotations, yet the description provides only a minimal overview. It omits crucial contextual details such as how the task is executed, the meaning of key parameters, and the workflow implications (e.g., using get_task to poll). The description is too sparse for the tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is only 25% (wait and model have descriptions), and the tool description adds no parameter meaning. It does not explain what 'prompt' or 'source_video_url' are for, nor mention parameters like 'timeout_ms' or 'callback_url'. The description fails to compensate for the sparse schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Create a Luma task') and the resource ('modify video'), and clarifies the platform ('on RunAPI'). It also mentions the return value, making it distinct from sibling tools like get_task (which retrieves) and check_pricing (which checks costs).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for modifying a video via RunAPI, but does not explicitly state when to use this tool versus alternatives like get_task or check_pricing. It gives context but lacks explicit exclusions or alternative guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_pricingA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so description carries full burden. It only states purpose, omitting behavioral traits like being a read-only operation, side-effects, or rate limits. With no extra disclosure, a score of 2 is appropriate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, perfectly front-loaded with the action and resource. No filler words; every word adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple lookup tool with 2 optional enum parameters and no output schema, the description is adequate. It fully captures the purpose. Could mention return value (pricing data), but not required given low complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, providing clear enum definitions for both parameters. The description adds no additional meaning beyond the schema, so baseline 3 is correct.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Look up') and the specific resource ('RunAPI pricing for the luma model line'). It distinguishes from siblings: 'get_task' retrieves task status, 'modify_video' modifies video content. No ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use when pricing info is needed, but provides no explicit guidance on when to use vs. the sibling tools 'get_task' or 'modify_video'. No exclusions or alternatives mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_taskA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the transparency burden. It clearly conveys a read-only fetch operation, but does not disclose potential caveats such as whether the tool blocks until completion, may return empty payloads, or requires authentication. The language is truthful but minimal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no filler. It effectively communicates the core purpose without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has a simple schema and no output schema, so the description must clarify return expectations. It mentions status and result payload, but lacks detail on possible states (e.g., pending vs. completed) or payload format, leaving some ambiguity for the agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters. The description adds no extra meaning beyond identifying a 'luma task', which maps directly to task_id. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool fetches the current status and latest result payload for a task, using a specific verb ('fetch') and resource ('luma task'). This distinguishes it from siblings like login, modify_video, and check_pricing, which serve different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Usage is implied: you call this tool when you need to check the status or retrieve the result of a previously created Luma task. However, there is no explicit guidance on when to use it versus alternatives, nor any mention of lifecycle prerequisites such as logging in first.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• loginA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It mentions browser flow and file saving but doesn't clarify conditional behavior based on existing credentials or force parameter.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, no wasted words, front-loaded with purpose. Efficient and clear.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simplicity (1 optional param, no output schema), the description covers key aspects but could mention when login is skipped.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% for the only parameter. The tool description adds no additional meaning beyond the schema's description of 'force'.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Authenticate RunAPI' with a specific mechanism (PKCE login flow) and saving location. It distinguishes from siblings which are about pricing, tasks, and video.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use for authentication but does not explicitly state when to use vs. alternatives or exclude scenarios. Context from siblings helps but not explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.