Server Quality Checklist

Tool Scores

• get_metricsA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavioral traits. It correctly indicates a read-only operation (returning metrics), but it does not mention behavior when no run exists, nor does it address authentication or rate limits. The description is accurate but minimal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single clear sentence that front-loads the main purpose. Every word contributes, with no redundancy or unnecessary detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Though the tool has no parameters and a simple purpose, the description lacks detail on the return format (e.g., are the metrics presented as numbers in an object?) and fails to explain the concept of 'the last run' or how runs are defined. Siblings exist, so more context would help.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are zero parameters, and the schema coverage is 100% (all parameters described). The description adds no parameter information, but per guidelines, 0 parameters yields a baseline of 4. No additional parameter semantics are needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns aggregate metrics (LLM calls, tool calls, revisions) of the last run. It uses a specific verb 'Return' and identifies the resource 'aggregate metrics', distinguishing it from siblings like run_task and get_trace.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage after a run (by referencing 'the last run'), but it does not explicitly state when to use this tool versus alternatives like get_trace or run_task. No when-not-to-use or prerequisite information is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_traceA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description bears full burden. It correctly states the tool returns a trace, but lacks disclosure on edge cases (e.g., behavior if no run_task call exists), potential size of output, or any side effects. The description is basic but not misleading.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence that front-loads the key information. No superfluous words, every part earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (no parameters, no output schema), the description covers the essential purpose and relation to run_task. However, it could be slightly more complete by clarifying what 'trace' contains or error handling, but the minimal context is adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters and schema coverage is 100% (trivially). Per rubric, zero params yields a baseline of 4. The description adds no parameter information, but none is needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'return', the resource 'full agent/tool trace', and the specific scope 'of the last run_task call'. It effectively distinguishes itself from sibling tools: run_task executes a task, and get_metrics returns metrics.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies that this tool should be used after a run_task call, but does not explicitly state when to use it versus alternatives, nor does it provide guidance on prerequisites or error conditions if run_task has not been called.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_taskA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It describes the orchestration process and the safety hold for sensitive actions but lacks details on failure modes, side effects, or performance implications.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the purpose, and contains no wasted words. It effectively communicates the tool's function and key safety feature.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (multi-agent orchestrator) and lack of output schema, the description covers the pipeline and safety mechanism well. It could mention what the tool returns (e.g., a trace) but is otherwise complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% for the single 'goal' parameter, so baseline is 3. The description does not add any additional semantics beyond the schema's 'The task to accomplish'.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool runs a goal through a multi-agent orchestrator with a specific pipeline (planner, specialists, critic). It distinguishes from sibling tools like 'get_trace' and 'get_metrics', which are retrieval tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context on when to use the tool, including that sensitive actions require human approval. However, it does not explicitly state when not to use it or suggest alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.