Server Quality Checklist

Tool Scores

• draft_proofD
  1.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It hints at being non-destructive by saying 'without sending traffic,' but does not explain what drafting entails, side effects, or response behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is succinct but too sparse to be effective. It lacks necessary detail while also not being well-structured to convey core functionality.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema, no parameter descriptions, and a vague purpose, the agent has insufficient information to determine when or how to invoke this tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  All three parameters (engagement_id, card_id, playbook_id) have no descriptions in the schema or prose. Coverage is 0%, and the description adds no meaning beyond the parameter names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose2/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Describe an allowlisted proof' is vague; 'describe' is not a strong verb for the action, and 'allowlisted proof' is ambiguous. It does not clearly distinguish itself from sibling tools like execute_proof or draft_waves.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines1/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The only usage hint is a negative constraint ('without sending traffic'), which is insufficient. No positive conditions or comparisons to alternatives (e.g., when to use draft_proof vs execute_proof) are given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• execute_waveC
  2.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of disclosing side effects. 'Run one admitted wave' hints at a mutating action but does not state whether it is idempotent, what happens to the wave, what errors occur, or what the rate limit entails. The token bucket reference suggests throttling but lacks concrete behavioral details expected for an execute-style operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, dense sentence that front-loads the core action with no filler words. Every word contributes to the intended meaning, and it is brief. However, its extreme brevity sacrifices clarity—conciseness is not a substitute for explaining 'admitted' or the token bucket without further context.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Even though the tool is simple (2 string params, no output schema), the description fails to cover key aspects like return values, side effects, or the meaning of 'admitted' and 'per-host token bucket.' For a mutating tool with no annotations, more behavioral context is necessary. The lack of any output or error information makes it incomplete for an agent to call this safely.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description does not mention either 'wave_id' or 'engagement_id'. There is no explanation of how the parameters influence execution or what 'admitted' means for them. The description provides zero help in understanding parameter semantics, leaving the agent completely reliant on parameter names alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description provides a verb ('run') and a resource ('wave') with additional context about a token bucket, but the meaning of 'admitted wave' is jargon-heavy and unclear without domain knowledge. It does not clearly differentiate from the sibling 'execute_proof'—both suggest executing something. It is not a tautology, but it fails to concretely state what the tool does or what a 'wave' is.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no guidance on when to use this tool versus the siblings. It does not mention alternatives like 'execute_proof' or conditions under which a wave is 'admitted.' The token bucket hint implies rate limiting but does not explain when a user should call this versus other execution tools. No exclusions or prerequisites are given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• engagement_healthC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. 'Report' suggests a read-only style operation, but the description does not explicitly state that no state changes occur, does not mention auth requirements or side effects, and provides no detail about what 'registered' or 'pending' statuses mean. With no output schema, return behavior is also undisclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single compact sentence with no filler or repetition. It front-loads the verb and the key reported categories, which makes it easy to scan, although it is terse enough that it contributes to under-specification in other dimensions.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations, no output schema, and no parameter descriptions, this one-line description is not enough for a fully informed call. A no-argument health check is guessable, but the behavior of engagement_id, the meaning of 'registered,' and the relationship to sibling status tools like query_cards and state_impact are left unexplained.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description never mentions engagement_id. The agent cannot tell whether the optional parameter filters the report to one engagement, scopes the results, or is required for a valid call. The description adds no meaning beyond the raw property name in the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'Report' and names concrete resources: registered engagements, cooldowns, and pending waves. This makes the tool's core purpose clear and distinguishes it from the execution-focused siblings like execute_wave and begin_engagement, though it does not explicitly differentiate it from query-oriented siblings like query_cards.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no guidance on when to use this tool versus alternatives. The description states what it reports but does not mention prerequisites, exclusions, or a preferred context such as 'check status before executing a wave.' An agent would have to infer usage from the tool name and sibling names.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• state_impactC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses that this is a write operation ('Record'), but with no annotations and no output schema, that's all it reveals. It doesn't specify whether this creates a new record, updates an existing state, requires any authentication, or what happens on repeated calls. For a mutation tool, this is a substantial gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, about eight words, with no filler. It leads with the action and object, making it easy to parse and free of redundant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With five required parameters, no annotations, and no output schema, this description is too minimal to support correct invocation. It doesn't explain what a valid 'preconditions' string looks like, what 'hunter_why' is for, or what the tool returns. The agent would need to inspect external docs or guess.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate, but it only mentions two of the five required parameters (impact, preconditions) and doesn't explain formats, constraints, or how they relate. engagement_id, card_id, and hunter_why are absent from the description, and the schema only labels them as strings. This leaves the agent to guess at their meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a concrete verb ('Record') with a specific resource ('a card') and identifies the payload ('hunter impact_class and preconditions'). It distinguishes itself from the sibling tools, none of which address recording impact state. However, it introduces the term 'impact_class' that doesn't appear in the schema ('impact'), and omits the other required fields from the description.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is given on when to use this tool versus the siblings. It doesn't state prerequisites, whether it should be called before/after other tools like execute_proof or draft_waves, or any alternative to use instead. The only inference is from the verb 'record', but that's not enough.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• begin_engagementC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description does not disclose any side effects, permissions, or error behaviors. Without annotations, the description is insufficient to understand what happens when the tool is invoked.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence without unnecessary words. It is well-structured and easy to read.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool lacks an output schema and annotations, and the description does not mention what the response contains, possible errors, or any other context. It is insufficient for an agent to understand the full behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description mentions 'scope,' 'rate limits,' and 'proof flags' which partially map to parameters like scope_hosts and requests_per_second, but it does not explain the meaning or format of each parameter. The schema has no parameter descriptions, so the description does not compensate adequately.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Register') and the resource ('one engagement'), distinguishing it from siblings that focus on proof execution or health checks.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool over alternatives. It does not mention any preconditions or scenarios where this tool is appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• execute_proofC
  2.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description bears the full burden of behavioral disclosure, and it does reveal a meaningful precondition — an allowlisted proof and operator confirmation — implying an approval gate beyond the schema's surface, which is useful. However, it says nothing about side effects, return values, reversibility, or whether execution is long-running, a notable gap for an 'execute' tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with no filler and the preconditions are stated directly. It is concise to the point of thinness — the efficiency is real, but the brevity reflects under-specification rather than disciplined economy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given zero annotations, no output schema, and 0% schema description coverage, the description leaves critical information uncovered: the meaning of a 'proof', expected parameter values, and the outcome of execution. For a 5-parameter (4 required) tool, this is incomplete and would leave an agent uncertain how to invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description offers no compensatory explanation of card_id, session_a, session_b, playbook_id, or engagement_id, or how they interrelate. With five unannotated string parameters, the agent is left guessing at values and formats, which the description failed to address.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('run') and a resource ('allowlisted proof'), with the 'allowlisted' qualifier adding an authorization constraint that helps set context. However, it never defines what a 'proof' is or differentiates this from close siblings like execute_wave and draft_proof, leaving the agent to infer the distinction on its own.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description offers no guidance on when to choose this tool over its siblings, despite obvious ambiguity with execute_wave, draft_proof, and draft_waves. The 'Requires allow_safe_proof and operator_confirmed' line reads as a precondition rather than a usage context, and no alternatives or exclusions are named.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• second_lookC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description must carry all behavioral information. It only says 're-run' and 'bounded template scan,' which hints at a read-only operation but does not disclose side effects, auth requirements, rate limits, or return behavior. This is sparse coverage that leaves significant behavioral uncertainty.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, tight sentence with no filler. It is front-loaded with the core action ('re-run') and scope ('bounded template scan'). While it is not verbose, its brevity comes at the cost of missing crucial details, so it earns a 4 for clarity of structure but not a 5.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the absence of annotations, output schema, and parameter descriptions, the description is severely under-informed. It does not explain what an 'engagement' or 'card' is, what a 'template scan' yields, or how to interpret results. For a tool with two required parameters and no output schema, this is insufficient for an agent to call it correctly without additional context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0% (parameters have no descriptions), and the description does not compensate. It mentions 'single card URL,' implying card_id is a URL, but leaves engagement_id unexplained. The agent must infer parameter purpose and types from names alone, which is insufficient.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('re-run'), resource ('bounded template scan'), and object ('single card URL'), making the tool's core function clear. It distinguishes implicitly from siblings like execute_proof or execute_wave by emphasizing a 'second look' on a single card, but it does not explicitly name alternatives, so it falls short of full differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 're-run' implies a use case where a previous scan already occurred and a refresh is needed, offering some contextual guidance. However, there is no explicit mention of when to choose this tool over siblings, and no exclusions are stated. The guidance is present but implicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• query_cardsB
  3.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description must carry the full burden of behavioral disclosure. It does state a key behavior: informational and hardening cards are hidden by default, which tells the agent about default filtering. However, it does not mention whether the tool is read-only, any permission requirements, rate limits, or failure modes. The non-mutating nature of a 'query' is implied but not explicitly stated.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely compact at two short sentences, leading with the primary purpose. It avoids redundancy and wastes no words, though it could have used the available space to clarify parameters or usage since it is so brief.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no output schema and no annotations, the description is incomplete. It does not describe the return format, possible results, pagination, error cases, or what constitutes 'hunter-relevant'. The single behavioral note about default hiding is helpful but does not make the tool safely callable by an agent that needs to know what to expect or how to interpret the output.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must explain the parameters. It does not explain what engagement_id is or what include_noise does beyond default false. The text 'Informational and hardening are hidden by default' indirectly suggests include_noise might control showing those, but it never explicitly links the parameter to that behavior. The agent is left guessing about the meaning and usage of both parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the tool returns 'hunter-relevant cards', which is a clear verb (return/query) and resource (cards). It does not formally distinguish itself from sibling tools, but the action-oriented siblings (execute_proof, draft_waves, etc.) are clearly different, so the purpose is recognizable without ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this is a query tool for retrieving cards, but it gives no explicit guidance on when to use it versus alternatives. The note 'Informational and hardening are hidden by default' hints at the include_noise parameter, but it does not explicitly say 'use include_noise when you need these types of cards' or provide any exclusions relative to other tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• draft_wavesA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of behavioral disclosure. It usefully states that the tool does not execute the waves and specifies the wave order. However, it does not disclose whether the proposal persists, requires permissions, or has any side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single short, front-loaded sentence with no filler. Every word contributes meaning, and the core distinction ('without executing them') is stated directly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple with only two required scalar parameters and no output schema, so the description does not need much. It covers the main purpose and non-execution, but it omits what the proposal produces or returns and how the parameters relate to the waves.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides only names and types with no descriptions, and the description never mentions the parameters. 'seed_url' and 'engagement_id' are somewhat self-explanatory, but the 0% schema coverage is not compensated by any parameter-level guidance in the description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('propose'), identifies the resource ('waves'), and names the exact wave sequence ('probe_alive then template_scan'). The phrase 'without executing them' clearly distinguishes this tool from the sibling execute_wave.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use when you want to preview or plan waves before execution, and 'without executing them' effectively rules out execute_wave. It does not explicitly name alternatives or state when to switch to execution, but the intended context is reasonably clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.