Server Quality Checklist

Tool Scores

• get_event_artifactC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden but only states it's a read operation. It doesn't disclose behavioral traits like authentication requirements, rate limits, error conditions, what 'sanitized' means, or how the JSON Pointer field_path works. The mention of 'sanitized' hints at data transformation but lacks details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Extremely concise single sentence with zero wasted words. Front-loaded with the core purpose, and every element (verb, resource, parameters) earns its place. No structural issues despite the brevity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 5 parameters with 0% schema coverage and no annotations, the description is incomplete—it doesn't explain parameter semantics or behavioral context. However, the existence of an output schema reduces the need to describe return values. For a read operation, this is minimally adequate but leaves important gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate but only lists parameter names without explaining their meaning. It mentions 'run_id, seq, and JSON Pointer field_path' but doesn't clarify what these identifiers represent, what 'offset' and 'limit' do, or how field_path syntax works. This leaves significant gaps in parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Read') and resource ('sanitized event artifact') with specific identifiers (run_id, seq, field_path). It distinguishes from siblings like get_run or list_runs by focusing on artifacts rather than runs themselves. However, it doesn't explicitly contrast with poll_events which might also retrieve event data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives like poll_events or get_run. The description mentions parameters but doesn't provide context about appropriate use cases, prerequisites, or when other tools might be more suitable. This leaves the agent without clear decision criteria.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_runsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden but discloses minimal behavioral traits. It indicates this is a listing operation but doesn't describe pagination, sorting, default behavior, rate limits, or authentication needs. For a tool with 3 parameters and no annotation coverage, this is inadequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's purpose without unnecessary words. It's appropriately sized and front-loaded with the core functionality.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has an output schema (which handles return values), the description's minimalism is partially acceptable. However, with 3 parameters, no annotations, and 0% schema coverage, the description should provide more context about filtering behavior and usage scenarios to be truly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate but adds no parameter information. It doesn't explain what 'status', 'backend', or 'cwd' mean, their relationships, or how they filter results. With 3 parameters (2 with enums) undocumented, this is a significant gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('List') and resource ('runs'), and specifies the scope ('known to the current orchestration MCP process'). However, it doesn't explicitly differentiate from sibling tools like 'get_run' or 'poll_events', which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'get_run' (for single run details) or 'poll_events' (for event monitoring). It mentions the scope but offers no explicit when/when-not instructions or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• cancel_runB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the action ('cancel') but doesn't explain what cancellation entails (e.g., whether it's reversible, if it stops processes immediately, what permissions are required, or any side effects like resource cleanup). For a mutation tool with zero annotation coverage, this is a significant gap in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence that directly states the tool's purpose without unnecessary words. It's front-loaded with the key action and target, making it efficient and easy to parse. Every word earns its place, with no redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (a mutation with one parameter) and the presence of an output schema (which handles return values), the description is minimally adequate. However, with no annotations and low schema coverage, it lacks details on behavioral aspects like side effects or error conditions. It meets the basic requirement but leaves gaps in understanding the tool's full context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has one parameter ('run_id') with 0% description coverage, meaning the schema provides no semantic details. The description doesn't add any information about this parameter (e.g., what a 'run_id' is, how to obtain it, or format examples). However, with only one parameter, the baseline is higher; the description implies the parameter identifies the run to cancel but doesn't compensate for the lack of schema details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('cancel') and the target ('a running external coding-agent run'), which is specific and unambiguous. It distinguishes from siblings like 'continue_run' or 'get_run' by focusing on termination rather than continuation or retrieval. However, it doesn't explicitly differentiate from all siblings (e.g., 'spawn_run' is about creation, but this isn't stated).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., that the run must be active), exclusions (e.g., not for completed runs), or comparisons to siblings like 'continue_run' for ongoing runs or 'list_runs' for status checks. This leaves the agent to infer usage context from the tool name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• continue_runB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. While it correctly indicates a write operation ('send'), it lacks critical details: required permissions, whether this changes run state, rate limits, error conditions, or what happens after sending input. For a mutation tool with complex nested parameters, this leaves significant behavioral gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose without unnecessary words. Every word earns its place, making it maximally concise while still conveying the essential action and context.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (2 parameters with nested objects), no annotations, but presence of an output schema, the description is minimally adequate. The output schema reduces need to describe return values, but the description should provide more context about run states, prerequisites, and parameter meanings to be truly complete for this interactive operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate but provides no parameter information. It doesn't explain what 'run_id' refers to, the structure of 'input_message', valid roles, or part types. With 2 parameters including complex nested objects, the description fails to add any semantic value beyond what the bare schema provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('send an additional input message') and target ('to a run that is waiting for more input'), providing specific verb+resource. However, it doesn't explicitly differentiate from sibling tools like 'spawn_run' or 'cancel_run', which would require mentioning it's for existing runs in a waiting state rather than creating new runs or terminating them.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context by specifying 'to a run that is waiting for more input', suggesting this tool should be used when a run is in a specific state. However, it doesn't provide explicit when-not-to-use guidance or name alternatives like 'spawn_run' for new runs or 'cancel_run' for termination, leaving some ambiguity about sibling tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_runB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool retrieves a 'summary status,' implying a read-only operation, but doesn't specify whether it's safe, if it requires authentication, rate limits, or what happens if the run_id is invalid. For a tool with no annotations, this is a significant gap in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence with zero waste: 'Get the current summary status for a known run.' It is front-loaded and efficiently conveys the core purpose without unnecessary elaboration, making it highly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (1 parameter, no nested objects) and the presence of an output schema (which handles return values), the description is minimally adequate. However, with no annotations and incomplete parameter semantics, it lacks details on behavioral traits and usage context, making it only partially complete for effective agent use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 1 parameter (run_id) with 0% description coverage, meaning the schema provides no semantic details. The description adds minimal context by implying 'run_id' refers to 'a known run,' but doesn't explain format, examples, or constraints. This partially compensates for the schema gap, but not fully, aligning with the baseline for moderate coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Get the current summary status for a known run.' It specifies the verb ('Get'), resource ('summary status'), and scope ('for a known run'), which is specific and actionable. However, it doesn't explicitly differentiate from sibling tools like 'list_runs' or 'poll_events', which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It mentions 'a known run' but doesn't clarify prerequisites (e.g., that the run must already exist from a previous operation) or contrast it with siblings like 'list_runs' (for listing runs) or 'poll_events' (for monitoring events). This leaves the agent without explicit usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• spawn_runB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. 'Start a new external coding-agent run' implies a write/mutation operation, but the description doesn't disclose permission requirements, rate limits, whether this consumes resources, what happens to previous runs, or how to monitor the started run. 'Return immediately' suggests asynchronous execution but lacks details about how to track completion or retrieve results.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise - a single sentence that communicates the core action and key behavioral trait ('return immediately'). There's zero wasted language, and it's front-loaded with the essential information. Every word earns its place in this minimal description.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a complex tool with 11 parameters, nested objects, and no annotations, the description is minimal. While an output schema exists (reducing need to describe return values), the description doesn't address the tool's role in the broader workflow with sibling tools, doesn't explain the asynchronous nature hinted by 'return immediately', and provides no context about error conditions or typical usage patterns. It's adequate but leaves significant gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 91% schema description coverage, the schema does most of the parameter documentation work. The description adds no parameter-specific information beyond what's in the schema. It doesn't explain relationships between parameters (like how 'session_mode' interacts with 'session_id'), typical values, or usage patterns. The baseline of 3 is appropriate given the high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Start a new external coding-agent run') and resource ('coding-agent run'), with the specific behavioral detail 'and return immediately' distinguishing it from potentially blocking operations. However, it doesn't explicitly differentiate from sibling tools like 'continue_run' or 'get_run' beyond the 'start new' aspect.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides minimal guidance - only that it starts a new run and returns immediately. There's no explicit guidance on when to use this versus alternatives like 'continue_run' for resuming sessions, 'cancel_run' for stopping runs, or 'poll_events' for monitoring. No prerequisites, constraints, or typical use cases are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• poll_eventsB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It mentions 'Long-poll' and 'incremental events,' hinting at real-time behavior and potential waiting, but lacks details on rate limits, authentication needs, error handling, or what 'events' entail. This leaves significant behavioral gaps for an agent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads key information ('Long-poll incremental events') without unnecessary words. Every part earns its place by specifying the action, resource, and key parameter.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 4 parameters with 0% schema coverage, no annotations, but an output schema exists, the description is moderately complete. It covers the core purpose and one parameter but misses details on other parameters, behavioral traits, and usage context. The output schema helps, but more guidance is needed for effective tool use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It explains 'after_seq' as 'after a known sequence number,' adding context beyond the schema's numeric constraints. However, it doesn't cover other parameters like run_id, limit, or wait_ms, leaving them undocumented. Baseline is 3 due to partial compensation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Long-poll incremental events') and resource ('for a run'), specifying it retrieves events after a known sequence number. It distinguishes from siblings like get_event_artifact or get_run by focusing on incremental polling rather than direct fetching, though it doesn't explicitly name alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for monitoring incremental events in a run after a specific sequence, suggesting it's for real-time updates. However, it doesn't explicitly state when to use this vs. alternatives like get_event_artifact or list_runs, nor does it mention prerequisites like needing an active run.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.