Server Quality Checklist

Tool Scores

• anb_redactA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully discloses: read-only, idempotent, no side effects, no external dependencies, and the over-redact guarantee. This provides high transparency for safe invocation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences: core function, behavioral traits, usage guidance. No fluff, front-loaded, every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with one parameter and an output schema (implied), the description covers purpose, safety, usage context, and behavioral guarantees. No gaps given the tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Only one parameter 'text' with 100% schema description coverage. The description repeats the same information as the schema ('known secret values and high-entropy tokens become <agent-vault:key> placeholders'), adding no new semantic value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action ('Scrub text') and resource ('text'), with clear output format ('<agent-vault:key> placeholders'). It is distinct from siblings which involve execution, listing, rendering, or status.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description advises 'Use before logging or returning any text that might contain a secret.' It also notes 'needs no Bob connection' and fail-safe behavior. While it doesn't explicitly list when not to use, the guidance is clear and helpful.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• anb_listA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description fully discloses behavioral traits: read-only and idempotent, no side effects; requires enrolled identity and reachable, unlocked Bob KMS daemon; errors on unreachable/locked or unauthorized identity. This compensates for the lack of annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loading key information (purpose, constraints, usage hint), with no unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple tool (0 params, output schema present), the description covers purpose, constraints, prerequisites, error conditions, and usage hints comprehensively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With zero parameters and 100% schema coverage, the baseline is 4. The description does not need to add parameter info, and it does not add any beyond what the schema provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists secret key names and metadata, never values, making the purpose specific and differentiating it from sibling tools like anb_exec and anb_render_to_file.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear usage context: use to discover key names before referencing them in anb_exec or anb_render_to_file, but does not explicitly state when not to use or list alternative tools for other scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• anb_render_to_fileA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description fully discloses behaviors: writes to disk (mode 0600, overwrites), requires identity and Bob, never returns content, and rejects absolute paths/.. traversal. No contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences efficiently convey purpose, side effects, prerequisites, and constraints. Front-loaded with action and key details, no superfluous information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the output schema exists (not shown but indicated as true), the description covers all necessary aspects: purpose, side effects, prerequisites, input constraints, and behavior. No gaps identified.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, but the description adds value: clarifies out_path as relative to render dir with security constraints, and explains template placeholders. This is beyond what the schema provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool renders a template with placeholders and writes to a file, with details on security and output. It explicitly contrasts with siblings by mentioning template rendering and file writing, distinguishing it from anb_exec, anb_list, anb_redact, and anb_status.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains when to use (render and write secrets to a file), mentions side effects (writes file, overwrites), and prerequisites (enrolled identity, unlocked Bob). It does not explicitly state when not to use or provide alternatives, but the sibling list implies context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• anb_execA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It thoroughly discloses side effects: 'spawns a real subprocess', 'default-deny', 'NOT idempotent (effects depend on the command)', requires Bob, 'Returns the exit code plus REDACTED stdout/stderr', and 'the raw secret is never returned even if the child prints it'. It also covers failure modes (denied or failed command returns non-zero exit_code with reason).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the main purpose and key constraints. It could be slightly more concise, but every sentence adds meaningful information. The length is justified given the tool's complexity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (execution with secrets, allowlist, prerequisites), the description is complete. It covers behavior, side effects, prerequisites, return values, and failure modes. An output schema exists, but the description still explains redaction and return structure adequately.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% (all three parameters have schema descriptions). The description adds value by explaining the KEY=VALUE form for env, placeholder resolution via Bob, no shell parsing for args, and absolute path requirement for command. This goes beyond the schema's formal definitions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Run an operator-allowlisted command with named secrets injected into the child process's environment.' It uses a specific verb ('run') and resource ('operator-allowlisted command'), and distinguishes itself from sibling tools like anb_list (listing) and anb_redact (redacting) by being the execution tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance: 'default-deny, only commands matching a scope=mcp allowlist rule run', 'NOT idempotent', 'Requires an enrolled identity and a reachable, unlocked Bob'. It tells when to use (allowlisted commands) and when not to use (if command not allowlisted or prerequisites not met).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• anb_statusA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so the description fully carries the burden. It discloses it is read-only, idempotent, returns no secret values, and has no side effects. This provides complete behavioral transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is four sentences, each adding essential information. It is front-loaded with the key purpose. No extraneous words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no inputs and an output schema, the description is complete: it covers behavior, error handling, usage context, and safety. The output schema exists, so return values are not needed in the description.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are no parameters, so the baseline is 4 per the zero-params rule. The description adds meaning by explaining what the tool does and what it returns, which is beyond the empty input schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description starts with 'Read-only self-check' which clearly specifies the verb and resource. It lists specific reports such as enrollment, client-cert presence, Bob reachability, etc. It also distinguishes this tool from siblings by stating it should be called first before anb_list, anb_exec, and anb_render_to_file.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly tells when to use: 'Call this first to confirm the vault is ready before...' and implies when not to use by mentioning alternatives. It also explains that on failure, the error field explains why, guiding the agent on next steps.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.