Server Quality Checklist

Tool Scores

• pi_abortB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It usefully discloses that the session remains alive and messages can still be sent, but it does not explain what happens to the aborted operation, whether the action is idempotent, or any side effects on the task's state.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two short sentences with no filler. The action is front-loaded, and the key consequence of using the tool (session stays alive) is stated immediately and clearly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter control tool, the description covers the most important distinction from sibling tools, but it omits parameter semantics and what happens after the abort completes. Since there is no output schema, more detail about the response or state would improve completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description does not mention taskId at all. The schema only says it is a required string, so the agent receives no additional meaning about what the parameter represents or how it should be obtained.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Abort') and resource ('the pi task's current operation'), and immediately differentiates from siblings by noting the session stays alive, so it is clearly not a full session teardown like pi_close. This makes the tool's purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context: use this to abort the current operation while keeping the session alive. However, it does not explicitly state when to prefer this over alternatives, nor does it mention when not to use it, leaving the guidance mainly implied.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• pi_closeA
  3.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It explicitly states the termination effect, registry removal, and the persistence of session files, which is valuable behavioral context beyond the tool name. However, it does not disclose irreversibility, permission requirements, or error conditions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two short sentences that immediately state the primary action and a key caveat. Every word adds information and there is no padding or repetition of the tool name.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is relatively simple with a single parameter and no output schema, but the description lacks guidance on when to use it versus siblings like pi_abort. It covers the main behavioral aspect (session files persist) but omits potential side effects or the tool's return behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0% and the description does not mention 'taskId' at all. While it is inferable that the parameter identifies which task to close, the description offers no explicit clarification of the parameter's format, meaning, or constraints, leaving the agent to guess.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly specifies the action ('Terminate'), the resource ('the pi process'), and the additional effect ('remove the task from the registry'), which distinguishes it from generic close/stop tools. The note that 'Session files stay on disk' further clarifies its scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like pi_abort or pi_respond. It does not mention any prerequisites, conditions, or scenarios that would make this tool preferable.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• pi_statusA
  3.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses the returned fields and the special 'pendingRequest' condition, providing useful behavioral context. However, it does not explicitly state read-only behavior or error cases.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is only two sentences, front-loaded with the purpose ('Lightweight status'), and contains no filler or redundant content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple read-only status tool with one parameter and no output schema, the description covers the main returned fields and a special case. It lacks detail on possible state values or error behavior, but is reasonably complete for its simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has one parameter taskId with no description, and schema description coverage is 0%. The description implies taskId identifies a 'pi task' but does not explicitly explain what the parameter is for, how to obtain it, or its format.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('status') and resource ('pi task'), and lists the exact information returned (state, streaming flag, last activity, etc.). This clearly distinguishes it from sibling tools like pi_dispatch, pi_read, and pi_list.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not provide explicit guidance on when to use this tool versus alternatives. It mentions 'lightweight' but never states when to prefer it over pi_read or pi_list, nor does it mention any exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• pi_listA
  3.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must shoulder the burden of behavioral disclosure. Simply stating the action (list) does not confirm it is read-only, nor does it mention potential large output volumes, pagination, or any safety guarantees. The minimal wording leaves the agent unsure about side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, focused sentence with no filler or redundancy. It front-loads the purpose and immediately conveys the tool's function.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter listing tool, the description covers the essential purpose and output content. However, without an output schema or additional notes, it does not specify ordering, pagination, or the exact structure of the returned list, leaving minor ambiguities.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  This tool takes zero parameters, making schema coverage trivially 100%. The description does not need to explain parameters, but it does identify the resulting data (tasks and states), adding a bit of output context. Given no params, a baseline of 4 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'List all pi tasks and their states' clearly names the verb (list) and resource (pi tasks), and specifies the output trait (states). It is distinct from sibling tools like pi_dispatch or pi_send, which imply different actions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies a global overview of all tasks, but does not explicitly state when to use this versus pi_status or pi_read, which likely handle individual task lookups. No exclusions or alternative references are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• pi_readA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the behavioral disclosure burden. It communicates that the tool is a read operation with cursor-based incremental semantics, but it does not clarify whether events are consumed/destroyed, what the response format is, or any defaults. This leaves meaningful gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no filler. It front-loads the core purpose and then provides the key usage hint about 'since', making it efficient and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has a simple parameter set and no output schema, so the description needs to convey return expectations and behavior. It gives examples of event types and a usage hint, but it does not state what the read returns, whether events persist after read, or how limit behaves. This is adequate but incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has no parameter descriptions, so the description must compensate. It explicitly explains 'since' as the latest seen sequence for incremental reads, which adds value beyond the schema. However, 'taskId' and 'limit' receive no additional context, leaving their semantics only implicit.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Read') and resource ('buffered events of a pi task'), with concrete examples of event types. This distinguishes it from sibling tools like pi_dispatch, pi_status, and pi_wait, which serve different actions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear usage context by explaining the incremental read pattern with 'since' as the latest sequence number. It does not explicitly mention alternatives or when-not-to-use scenarios, so it falls short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• pi_respondA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It does disclose the behavioral meaning of the optional parameters (value for select/input/editor, confirmed for confirm, cancelled to dismiss) and adds context from pi_status/pi_wait. However, it does not explain side effects, whether the request is consumed, error conditions, or what happens after responding, leaving partial transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, information-dense sentence that front-loads the action and object. Every clause earns its place by explaining the trigger and the parameter semantics, with no filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the core workflow (respond to a pending dialog) and parameter modes, but it leaves gaps in required context: taskId is unexplained, and there is no mention of return values or error behavior. With no output schema or annotations to fill these gaps, the description is adequate but not fully complete for a 5-parameter tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 80%, covering value, confirmed, cancelled, and requestId. The description adds the practical mapping of these parameters to dialog types, which is helpful, but it does not explain the required taskId parameter or its relationship to the pendingRequest. It does reference the source of requestId via pi_status/pi_wait, but that is already implied by the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states what the tool does: 'Answer a pending UI dialog reported as pendingRequest by pi_status/pi_wait.' It uses a specific verb ('Answer'), identifies the resource ('pending UI dialog'), and ties the tool to a concrete source (pi_status/pi_wait), distinguishing it from siblings like pi_send or pi_abort.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides clear context for when to use the tool: when a pendingRequest is reported by pi_status/pi_wait. It also explains the three usage modes (value, confirmed, cancelled), which serve as conditional guidance. However, it does not explicitly mention alternative tools or say when not to use this tool, so it stops short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• pi_dispatchA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden and does disclose key behaviors: the subprocess spawn, the working directory, immediate async return, and the need for pi_wait/pi_read to retrieve results. It does not cover failure modes or resource cleanup, but the core behavioral traits are clearly revealed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the action and mechanism, then provides workflow guidance. Every word contributes, with no repetition or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 6 parameters but all are fully schema-documented, and the description covers the essential dispatch workflow, including the immediate return and follow-up tools. It lacks explicit error-handling details, but for a straightforward dispatcher with rich schema coverage, this is sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline 3 applies. The description adds little parameter-specific meaning beyond what the schema already provides; it only mentions cwd indirectly and taskId as a return value, not as a parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb+resource ('Dispatch a task to a new pi sub-agent session') and clearly states the mechanism (spawns `pi --mode rpc`). It differentiates from sibling tools by noting the immediate return of a taskId and explicitly referencing pi_wait and pi_read for follow-up.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides clear context on when to use this tool (to start a new session) and directs the agent to pi_wait for blocking and pi_read for progress inspection. However, it does not explicitly contrast with pi_send or state when not to use this tool, such as for existing sessions or non-blocking sends.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• pi_waitA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the transparency burden. It details the blocking behavior, exit conditions, and return fields. It stops short of noting side effects or auth requirements, but the wait-and-return semantics are clearly disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences front-load the purpose and then list return values. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description explains return values despite no output schema, and covers the main blocking behavior and the special pendingRequest condition. It doesn't address error cases, but for the tool's apparent simplicity, this is sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema covers only timeoutMs with a description; taskId lacks meaning. The description references 'timeoutMs elapses' and 'pi task' behaviorally, but doesn't elaborate on parameter formats or constraints. For 2 parameters with 50% coverage, this is adequate but not enriching.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Block until the pi task's current run finishes' – a specific verb+resource that clearly identifies the tool's function. It distinguishes it from siblings like pi_status by emphasizing blocking behavior and return of final state, summary, and pendingRequest.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear context: it blocks until a run finishes, a dialog needs answering, or timeoutMs elapses. It doesn't explicitly name alternatives, so it's not a 5, but the conditions imply when to use it, especially the note about waiting for pi_respond.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• pi_sendA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden and explains mode-specific behaviors: auto switches between direct prompt and steering based on activity, and followUp queues until the agent stops. It lacks details on errors or return values but adequately discloses key behavioral traits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, with only two sentences. The first sentence states the core purpose, and the second explains modes without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the essential behavior and mode selection, which is sufficient for a message-sending tool. It does not describe return values or explicitly position vs siblings, but the core context is complete enough for an agent to use it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, but the description adds meaningful semantics for the 'mode' parameter by explaining 'auto' and 'followUp'. The required parameters taskId and message are inherently clear from the description, so the tool compensates for the schema gap effectively.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Send a message to a pi task' uses a specific verb and resource, clearly distinguishing the tool's function. It also introduces mode behavior, adding depth beyond a simple send action.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides context on when to use different modes: 'auto' for idle or working states, and 'followUp' to queue until the agent fully stops. It does not explicitly compare to sibling tools, but the mode guidance is clear and actionable.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.