Server Quality Checklist

Tool Scores

• vault_kv_readA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already provide readOnlyHint, destructiveHint, idempotentHint, and openWorldHint. The description adds behavioral context beyond annotations: versioning support for KV v2, and auto-detection of kv_version. No contradictions. Minor missing details like error handling or auth requirements.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with purpose. Every sentence adds value with no fluff. Highly concise and clear.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With an output schema present (not shown but indicated), explanation of return values is unnecessary. The description covers versioning, mount/path format, and kv_version auto-detection. Missing error behavior but acceptable for a read operation. Completeness is high given available context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so baseline is 3. The description adds value by explaining version parameter semantics ('specific historical version; omit for latest') and clarifying mount/path formatting. This extra context justifies a 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Read') and resource ('key/value data stored at a secret path in a Vault KV engine'). It distinguishes between KV v1 and v2 regarding version, and the tool is distinct from siblings like vault_kv_list and vault_kv_read_metadata.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains when to use the version parameter for KV v2 but does not explicitly guide the agent on when to choose this tool over siblings (e.g., vault_kv_read_metadata for metadata, vault_kv_list for listing). Implicit context is provided but no clear when-not or alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• vault_kv_read_metadataA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and destructiveHint=false, so the description's job is to add context. It does so by listing what metadata is returned (version history, times, custom_metadata), providing behavioral detail beyond the annotations. No contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that is front-loaded with the critical 'KV v2 only' qualifier. Every word adds value; no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the presence of sibling tools, annotations, and an output schema (which documents return values), the description covers the essential purpose, scope (KV v2 only), and included data. It does not mention error conditions or authentication, but those are likely standard for Vault tools. Overall sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, and the schema already describes both parameters clearly. The description does not add additional parameter-level meaning beyond what the schema provides. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool reads metadata for a secret in KV v2, listing specific data fields (version history, current version, timestamps, custom_metadata). It distinguishes from siblings like vault_kv_read (data) and vault_kv_list (listing paths) by emphasizing it's for metadata only.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly limits usage to KV v2 ('KV v2 only'), which helps agents choose between this and vault_kv_read for data. However, it does not explicitly state when not to use or name alternatives, but the sibling context makes the distinction clear enough.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• vault_list_kv_mountsA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already provide safety and idempotency info. Description adds that it returns KV mount versions, but no further behavioral traits beyond what annotations indicate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences, immediately clear purpose, no fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given zero parameters and clear annotations, description is complete. Output schema handles return values.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters exist, and schema coverage is 100%. Description doesn't need to add parameter details; baseline 4 for zero-parameter tools.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states it discovers secret engine mounts and their KV versions, differentiating from sibling tools that operate on specific paths or health checks.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states utility: learn mount values for other tools. Implicitly when-not-to-use is not needed as it's a discovery tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• vault_healthA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate read-only, non-destructive, idempotent. Description adds specific health checks (init, seal, standby), enhancing transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences, front-loaded with purpose. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Tool is simple with 0 parameters and output schema present. Description is complete for its function.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters exist, so baseline 4 applies. Description need not add param info.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool checks Vault server health, specifying initialized, sealed, and standby status. It distinguishes from sibling KV tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description says 'Good for verifying connectivity and configuration,' providing clear context. No exclusions needed as siblings are unrelated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• vault_kv_listA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, destructiveHint, idempotentHint. The description adds behavioral context: keys ending in '/' are sub-folders, and kv_version auto-detection can be bypassed. This adds value beyond annotations without contradiction.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences that are front-loaded and free of fluff. First sentence states exact purpose, second sentence adds key nuance. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With all parameters documented in schema, behavioral details provided, and an output schema present, the description covers everything needed for an agent to correctly invoke this tool. No gaps identified.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so baseline is 3. The description adds extra meaning: explains path behavior for root listing and kv_version's performance benefit. This enriches understanding of parameters beyond schema alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists secret keys/sub-folders under a path, with a specific verb and resource. It distinguishes from siblings like vault_kv_read and vault_kv_read_metadata by focusing on listing versus reading. Additional details (keys ending in '/') add clarity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains how to list the root using path '' or '/', providing clear usage context. However, it does not explicitly state when not to use this tool or name alternatives, though sibling tool names are provided in context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.