Server Quality Checklist

Tool Scores

• submit_script2videoC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It only mentions the asynchronous nature (returns immediately, poll for progress) but omits other critical traits like destructiveness, authentication needs, rate limits, or error states.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loads the action ('Submit'), and wastes no words. It is efficiently structured for quick comprehension.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (5 undocumented parameters, no output schema, no annotations), the description fails to provide sufficient context. It does not explain return values beyond job_id, parameter roles, error handling, or lifecycle details, leaving large gaps for an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 5 parameters with 0% coverage (no descriptions), and the tool description does not explain any parameter meanings or defaults. Only 'script' is implied by the tool name, but parameters like 'user_requirement', 'style', 'profile', and 'job_id' are left completely undefined.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool submits a script-to-video job and returns a job_id immediately, indicating a clear action. It differentiates from the sibling 'submit_idea2video' by specifying 'script' as input, though it does not explicitly contrast them.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides a use case: when you have a script to convert to video. It also advises polling 'get_job_status' for progress. However, it lacks guidance on when not to use this tool or alternatives like 'submit_idea2video', limiting its completeness.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_job_statusB
  3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided; description only lists what is returned (state, progress, error list) without disclosing read-only nature, error handling, or permission requirements.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  One concise sentence, front-loaded with action and resource; efficient but could be minimally expanded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite simplicity, lacks details on return format or potential states; with no output schema, description should provide more context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has 0% description coverage; description adds no extra meaning for the required job_id parameter (e.g., format, where to obtain it).

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states it retrieves current state, progress, and error list for a submitted job, distinguishing it from sibling tools like cancel_job or submit_*.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Implied usage when needing job status, but no explicit guidance on when to use or not use this tool versus alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_artifactsB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, and the description fails to disclose behavioral traits such as whether the job must be completed, pagination, order of results, or any side effects. The description only hints at the kind parameter.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is short and to the point, with a clear separation of purpose and parameter info. It could be improved with a more structured format, but it is not verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema and no annotations, the description is insufficient: it does not mention the return format, whether the job must be in a certain state, error handling, or limits on the number of artifacts listed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description adds value by enumerating valid kind values ('final | frames | intermediate | all'), but does not explain the job_id parameter beyond its necessity.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('List files') and the resource ('produced by a job'), and mentions the kind parameter with valid values, distinguishing it from sibling tools like cancel_job or get_job_status.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool vs alternatives, no prerequisites or context for the job_id parameter, and no explanation of the different kind options' use cases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• cancel_jobA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses that working_dir is preserved, which is a useful side effect. However, it omits details like required permissions, irreversibility, or whether the cancellation is immediate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two short sentences with no filler. Every word carries meaning. Highly concise and well-structured for quick parsing.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter, no output schema), the description adequately covers purpose and key behavioral detail (working_dir preservation). It could be slightly more complete by addressing the parameter format, but overall it is sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema lists a single required parameter (job_id) with no description. The tool description does not explain how to obtain or format job_id. Since schema description coverage is 0%, the description should compensate but does not add meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Cancel') and the resource ('a running or queued job'), with an additional note about working_dir preservation. This distinguishes it from sibling tools like get_job_status (status check) and submit_* (submission).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description tells when to use (for running or queued jobs) but does not provide explicit guidance on when not to use or mention alternatives. For example, it could advise checking status via get_job_status before canceling, or note that canceling is irreversible.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• submit_idea2videoA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Despite no annotations, the description discloses the asynchronous nature (returns job_id immediately) and the need to poll for completion. It does not discuss failure modes, idempotency, or auth requirements, but covers the core behavioral trait.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences with no extraneous information. The most important information (purpose and async behavior) is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema and 5 undocumented parameters, the description is insufficient. It doesn't specify the response format beyond 'job_id', nor does it explain parameter semantics or constraints (e.g., idea character limit).

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description does not explain any of the 5 parameters (idea, user_requirement, style, profile, job_id). The agent gets no help understanding what each parameter means or how to use them.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the verb 'Submit' and the resource 'idea-to-video job', differentiating it from sibling tools like get_job_status and cancel_job. The asynchronous behavior is also made explicit.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides context by telling the agent to poll get_job_status for progress, implying a two-step workflow. However, it does not explicitly state when not to use the tool or mention alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_quotaA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided. Description states it returns quota usage but does not disclose whether it's read-only, authentication needs, or any side effects. For a simple read tool, lacking explicit transparency is acceptable but not ideal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, no fluff. Clearly and concisely communicates the tool's purpose and scope.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description gives a good idea of what to expect (usage per provider). Could mention if it returns numbers or remaining amounts, but adequate for a simple quota check.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Zero parameters, schema coverage 100%. Description adds context by specifying the scope (today's, UTC day) and who it covers (chat/image/video providers), adding meaning beyond the empty schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Exactly states it gets today's daily quota usage for chat/image/video providers in UTC day. The verb 'get' and resource 'quota' are clear, and it distinguishes from sibling tools that handle job submissions and status.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Implied usage is to check quota before submitting jobs, but no explicit when-to-use or alternatives. No mention of when not to use or compared to sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.