Server Quality Checklist

Tool Scores

• execute_runbookA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds behavioral detail beyond the annotations by warning that only commands declared in a version-controlled runbook may be executed, and by disclosing both the global execution gate and the special approval token needed for mutating plans. This meaningfully helps an agent understand the operational restrictions around execution, while the destructiveHint annotation is not contradicted.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise: two sentences that front-load the core scoping rule, then state the indispensable prerequisites. Every sentence contributes meaningful information, with no repetition or irrelevant detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description adequately covers the approval and mutating-plan aspects of the tool, which matters given its destructive, non-idempotent nature. However, with three parameters, nested objects, and no output schema, it omits crucial guidance about what the 'id' refers to, how variables are used, and what happens on invocation. This leaves the definition minimally usable, not fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description itself must clarify parameters, but it does not. It indirectly references the approvalToken concept when mentioning an out-of-band approval token, but it never explains the 'id' parameter or the 'variables' object, both of which are non-obvious and important for actual invocation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a concrete verb, 'execute', and a specific scoped resource: commands declared only in a version-controlled runbook. It clearly differentiates this execution action from the sibling 'inspect_runbook' and 'list_runbooks' tools by focusing on execution rather than inspection or enumeration.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear preconditions for use: the global execution gate is required, and mutating runbooks additionally require an out-of-band approval token. It does not explicitly state when not to use the tool or compare it to siblings, but the prerequisites provide enough context for when it is valid to invoke.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• inspect_runbookA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the description doesn't need to restate those. The description adds value by explicitly stating 'without executing', which clarifies that no side effects will occur, and it lists what the tool returns (commands, plan hash, risk, approval requirement). This goes beyond the annotations, which only cover the general safety profile. The description doesn't contradict annotations; in fact, it reinforces them.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence that front-loads the action ('Validate variables and return the exact commands') and adds the critical constraint ('without executing'). Every word earns its place, and it's appropriately sized for the tool's complexity. There is no fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For an inspection tool with no output schema, the description covers the essential return values (commands, plan hash, risk, approval requirement) and explicitly says it doesn't execute, which is a key behavioral detail. It doesn't mention any security implications or prerequisites, but given the simplicity and annotations, it's fairly complete. The only minor gap is explaining what 'resolved' means, but it's not critical.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description does not add specific detail about the parameters. The 'id' parameter is obvious, but 'variables' is only hinted at by the phrase 'Validate variables', which gives some context. However, the description doesn't explain the structure of the variables object or any constraints, which the schema already provides some info on. With 0% coverage, the description should do more to compensate, but it partially does by mentioning 'validate variables'. This is a baseline 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the verb 'validate' and 'return' to specify what the tool does, with the resource being a 'resolved runbook plan'. It clearly enumerates the outputs (commands, plan hash, risk, approval requirement) and emphasizes 'without executing', which distinguishes it from its sibling 'execute_runbook'. However, it could be more explicit about the 'inspect' nature versus 'inspect' in the name, but it's clear enough.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'Validate variables and return the exact commands... without executing' clearly implies this tool is for inspection, not execution, and the title 'Inspect a resolved runbook plan' sets the context. It doesn't explicitly name the sibling 'execute_runbook' as the alternative, but the 'without executing' phrase effectively communicates the distinction. It doesn't mention when to use this over 'list_runbooks', but the purpose is distinct enough.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_runbooksA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false. The description adds value by saying the tool does not execute runbooks, which is a meaningful behavioral guarantee beyond generic read-only, and notes the resource is version-controlled.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two tight sentences with no filler. The primary action and result are front-loaded, and the critical behavioral clarification 'Does not execute anything' earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a no-parameter read-only list tool, the description covers what the call and returns conceptually: version-controlled runbooks and their risk levels. There is no return schema, but the description gives sufficient context for invocation and interpretation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so there is no parameter semantics gap to compensate for. The description needs no parameter documentation to make the tool invocable.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: 'List version-controlled runbooks and their risk levels.' The added 'Does not execute anything' clearly separates it from execute_runbook, and 'list' vs 'inspect' is self-evident against inspect_runbook.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'Does not execute anything' provides a practical exclusion and implicitly points to execute_runbook for execution needs. It does not explicitly mention inspect_runbook, but the list-vs-inspect distinction is clear enough from context and the sibling names.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.