Server Quality Checklist

Tool Scores

• vault_entry_getA
  4.1/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Given no annotations, the description fully discloses critical behavior: the secret is NEVER returned to the agent, processed server-side, and burned after reading. It also provides a directive to not store or reference the secret, ensuring transparency about side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences plus a directive, all relevant and front-loaded with the main action. It is efficient but could be slightly more streamlined without losing clarity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The core behavior, burn directive, and no-return policy are well covered. However, the optional command execution is mentioned but not explained how to specify the command (no param for it in schema), leaving a gap for the agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and both parameters have descriptions. The description adds only minor context ('from vault_entries_list' and 'Be specific' for purpose), which does not significantly enhance meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'retrieve', the resource 'vault secret', and key details about the burn directive and optional command execution. It distinguishes from sibling tools like vault_entries_list (list only) and vault_exec (execute command separately).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not explicitly state when to use this tool versus alternatives like vault_entry_inject_env or vault_exec. It mentions 'use it for the stated task only' but provides no when-not guidance or comparison with siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• vault_entries_listA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses that no secret values are returned, which is a key behavioral trait. No annotations exist, so description carries full burden; however, does not mention potential limits or pagination behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, no redundancy. Action and result are front-loaded; every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Adequately describes purpose, return content, and use with sibling. Lacks mention of output limits or sorting, but acceptable for a simple list tool with no parameters.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Tool has zero parameters with 100% schema coverage. Description adds no parameter details but clarifies return behavior, meeting the baseline for no-parameter tools.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the tool lists all vault entries, specifies it returns only entry IDs and secret names (no values), and distinguishes from sibling vault_entry_get by noting the need to store the entry ID for that tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides guidance on using the output with vault_entry_get, implying this tool is for discovery before retrieval. Lacks explicit when-not-to-use or alternatives, but the context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• vault_rsyncA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Since no annotations are provided, the description carries the full burden. It discloses a key behavioral trait: the SSH key is written to a temp file for the duration of the transfer and deleted immediately after. This is important security context. It does not mention other behavioral aspects like whether the operation is destructive or rate limits, but the key handling disclosure is sufficient for a score of 4.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise with two sentences, no wasted words. The first sentence packs the purpose and a key behavioral note, and the second sentence gives a usage recommendation. It is efficiently structured and easy to scan.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no output schema and 7 parameters, the description covers the essential inputs and key behavior. It could optionally mention what the tool returns (e.g., success/error or rsync output), but this is not critical. Overall, it provides sufficient context for an agent to use the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  All 7 parameters have schema descriptions (100% coverage), so the baseline is 3. The description adds meaningful extra context: the trailing slash behavior for local_path (contents vs directory sync) and notes extra_args as optional. This additional information warrants a score of 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool syncs a local directory to a remote host using rsync over SSH with a vault-fetched SSH key. It specifies the verb 'sync', the resource 'local directory to remote host', and the method 'rsync over SSH'. It also explicitly contrasts with the sibling tool vault_exec + python hex-encoding for deploying files, providing a clear distinction.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly advises to use this tool instead of vault_exec + python hex-encoding for deploying files to remote servers, giving clear usage context. However, it does not elaborate on when not to use it or other alternatives, which prevents a score of 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• vault_entry_forgetA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, but the description fully discloses that it is a no-op on the server and only affects local context. This is critical for understanding tool behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences, front-loaded with action and no wasted words. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given low complexity (1 param, no nesting, no output schema), the description covers purpose, behavior, and use case completely.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has 100% coverage with description for entry_id. The description adds meaning by explaining it's a reference to discard, not the actual secret, which is beyond schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Discard a stored vault entry reference', specifying the verb and resource. It distinguishes from sibling tools like vault_entry_get (retrieves) and vault_entry_inject_env (injects).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit use case: 'clean up stale references when a secret has been rotated or revoked.' Implicitly contrasts with server-side deletion, but no direct alternatives mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• vault_entry_inject_envA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description must carry burden. Clearly discloses that the secret is decrypted server-side and written directly, and that plaintext is never returned to the agent – critical security behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two highly informative sentences. Front-loaded with action. Zero waste – every sentence adds value (purpose, security behavior, usage advice).

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 4 required parameters, no output schema, and simple write operation, the description fully covers purpose, behavior, security, and usage guidance. No gaps for an AI agent to correctly invoke.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, baseline 3. Description adds no additional meaning beyond what schema already provides for each parameter. It gives overall context but no per-parameter enrichment.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states 'Write a vault secret directly into an environment variable file (.env)' – specific verb+resource+target. Explicitly distinguishes from sibling tool 'vault_exec' with 'Use this instead of exec file-writing commands.'

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit guidance on when to use this tool ('Use this instead of exec file-writing commands') and describes a key behavior (plaintext never returned). No ambiguity about context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• vault_execA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses critical security behaviors: secret never returned, buffer zeroed, injection via exec_config, and rejection of escape patterns. Explains remote SSH key handling without disk exposure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Efficiently structured with primary purpose first, then security, then execution details, and finally constraints. No superfluous sentences; every sentence contributes.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (6 params, nested objects, no output schema), the description thoroughly covers execution modes, security, injection, and constraints. No gaps for correct usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema already covers 100% of parameters with descriptions. The description adds value by explaining the overall flow, when parameters are optional, and how inject_as overrides work.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Execute a shell command with a vault secret injected as an env var.' It specifies the action, resource, and security context, distinguishing it from other vault tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit when-to-use, when-not (omit entry_id for remote-only), and prohibitions (no shell escape patterns). Also distinguishes between LOCAL and REMOTE execution modes.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.