Server Quality Checklist

Tool Scores

• cancelC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, and the description does not disclose side effects, irreversibility, or asynchronous behavior beyond the basic cancellation action. Agent receives minimal behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence is concise but lacks structure (e.g., bullet points) and is too brief to convey essential context for effective tool selection.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter tool without output schema, the description provides the essential purpose and state constraint, but omission of cancellation semantics (idempotency, error cases) leaves gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description does not add meaning to the job_id parameter; it neither explains what a job_id represents nor provides any format or example.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the action (cancel) and the resource (job), specifying 'running or pending' scope, which distinguishes it from sibling tools like start_orchestration or get_status.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidelines on when to use this tool versus alternatives, no mention of when not to cancel, and no reference to sibling tools for context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_artifactsB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It mentions a precondition but omits details like return format, error cases, and what constitutes 'final artifacts'. The behavioral disclosure is minimal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single efficient sentence with no waste. However, it may be too terse given missing behavioral and parameter details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema and only one sentence, the description fails to explain what 'final artifacts and file lists' entail or how the tool behaves in different states. It is insufficient for a complete understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% and the description adds no meaning for the 'job_id' parameter beyond its existence. The agent gets no information on format, constraints, or examples.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Return' and the resource 'final artifacts and file lists', which distinguishes it from sibling tools like get_status or approve_plan. It also adds a precondition (job must be done or cancelled) for further specificity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly requires the job to be done or cancelled before calling this tool, providing clear temporal context. It does not list alternatives but the sibling tools have distinct purposes, making the usage scope clear enough for an agent.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• approve_planB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description reveals that the tool modifies state (approve/replace) but lacks details on side effects, permissions, or irreversible consequences.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence that efficiently conveys the core action and key parameters, though 'or replace via edits' is slightly verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  No output schema or annotations; description omits prerequisites, error conditions, or return behavior, making it insufficient for a tool that affects multiple tenants.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description adds minimal parameter context—'job_id' is implied but 'edits' is not explained, and 'launch tenants' is ambiguous.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool approves or replaces a plan using 'job_id' and mentions 'launch tenants', distinguishing it from siblings like cancel or get_status.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for approving plans but provides no explicit guidance on when to use this vs alternatives like start_orchestration, nor any exclusions or preconditions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_statusB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of disclosing behavioral traits. It fails to mention that the operation is read-only (safe) or any potential side effects, auth requirements, or rate limits. The description only states what is returned, not how it behaves.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no redundancies. Every word contributes to the core purpose: stating the action, resources, and required parameter. This is an optimal length for a simple tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (1 parameter, no output schema, no annotations), the description covers the essential information about what it does and takes. However, it lacks details about the return structure (e.g., whether the response is a flat object or nested) and error cases, which would help the agent handle the output correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 1 parameter (job_id) with 0% description coverage. The description merely mentions 'job_id' without adding format, source, or example values. It does not compensate for the lack of schema-level documentation, leaving ambiguity about valid inputs.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Return' and the resources 'status, plan, and per-tenant states' for a given 'job_id'. This distinguishes it from siblings like 'approve_plan' or 'start_orchestration' which serve different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for retrieving status but does not explicitly state when to use versus alternatives (e.g., when to use get_artifacts instead). No guidance on prerequisites or exclusions, leaving the agent to infer context from the tool name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• start_orchestrationA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It discloses the main behavior (decompose and return plan awaiting approval) but does not mention side effects, permissions, or storage details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence of 13 words, front-loading the core action and result. It is appropriately sized with no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple schema and sibling tools, the description adequately covers the tool's role as the starting point. It mentions the returned values and state, but lacks explanation for the output_dir parameter.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% as the description does not mention any parameters. While parameter names ('Prompt', 'Output Dir') are somewhat intuitive, the description adds no value beyond the schema, failing to clarify optional parameters like output_dir.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states the tool's action: decompose a prompt into a plan, returning a job_id and plan. It clearly differentiates from sibling tools (approve_plan, cancel, etc.) which serve different stages of the workflow.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this is the initial step for orchestration, but does not explicitly state when to use it over siblings or mention prerequisites. The purpose is clear enough that an agent would know to call it to start an orchestration.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.