Server Quality Checklist

Tool Scores

• storeSecretC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It adds one useful detail ('The value is never returned') but omits critical traits such as overwrite behavior, error handling, permissions required, or whether the operation is idempotent. This is insufficient for a mutation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise, consisting of two short sentences that each add value. It front-loads the core action and appends a key behavioral note, with no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations, output schema, and parameter descriptions, the description is under-specified. It does not clarify whether storing an existing name overwrites or errors, how the 'project' parameter affects behavior, or what a successful call returns (beyond noting the value itself is never returned). This leaves significant gaps for safe usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description does not explain any of the parameters (name, value, project). It neither lists them nor describes their meaning, leaving the agent to rely solely on schema constraints like the name pattern and value minLength, which is not enough.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Encrypt and store a secret value.' It uses a specific verb ('store') and resource ('secret value'), and explicitly notes that the value is never returned, which distinguishes it from sibling tools like listSecrets and runWithSecrets.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like rotateSecret or deleteSecret. There is no mention of prerequisites, exclusions, or appropriate use cases, leaving the agent to infer based on the tool name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• rotateSecretB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided. The description discloses that the value is never returned, which is important security context, but it does not mention behavior for non-existent secrets, permissions, or what response to expect.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences with no filler, but it omits parameter details and usage context, making it efficient yet under-specified.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 3 parameters and no output schema, the description lacks essential operational details like error conditions, success indicators, and when rotation is applicable compared to storeSecret.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%. The description only mentions 'value' indirectly, with no explanation of 'name', 'project', or how they relate to the operation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'Replace' and identifies the resource 'secret's value', clearly distinguishing from siblings like storeSecret (store) and deleteSecret (delete).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for changing an existing secret's value but does not explicitly state when to use it versus storeSecret or provide any prerequisites or exclusion criteria.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• deleteSecretA
  3.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It discloses two important behaviors: the delete is recoverable (soft-delete) and the secret value is never returned. This goes beyond the name/title but could still mention permissions or error handling for a more complete picture.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two short sentences that are front-loaded with the core action. Every word adds value: the first sentence states what it does, the second adds an important behavioral constraint. There is no redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the primary purpose and a key constraint, but fails to provide context on when to use it, how recovery works, or how it relates to sibling operations like rotation. Given the simple parameter set and lack of annotations, the description is minimally adequate but leaves gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has one parameter 'name' with a pattern, but no description. The tool description does not explain what 'name' refers to or any additional semantics beyond the schema's pattern. Since schema description coverage is 0%, the description needed to compensate but did not.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'soft-delete' and identifies the resource as 'a secret', which clearly distinguishes it from sibling tools like listSecrets, storeSecret, rotateSecret, and runWithSecrets. The parenthetical '(recoverable)' adds meaningful differentiation from a permanent delete.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives no guidance on when to use this tool versus alternatives like rotateSecret or runWithSecrets. It implies deletion use, but does not state explicit conditions, prerequisites, or situations where a different operation would be more appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• listSecretsA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the transparency burden. It discloses a critical behavioral trait: it does not return secret values, which is important for a secrets management tool. However, it doesn't mention other behaviors like permissions, pagination, or return format.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two short sentences, with the key information front-loaded. It conveys the core action and the most important caveat without any fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description gives the basic action and a key caveat, but it lacks details about the return structure and the project parameter. Given no output schema and no parameter guidance, the description is only minimally complete for a simple list tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The single 'project' parameter has no schema description, and the tool description doesn't mention it at all. Since schema coverage is 0%, the description fails to convey whether project is a filter, a required context, or its format.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists secret names and metadata, with the explicit caveat that it does not return secret values. This distinguishes it from sibling tools that store, rotate, delete, or run with secrets.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives no explicit guidance on when to use this tool or how it compares to alternatives. The only hint is the exclusion of secret values, implying it's not for retrieving values, but it doesn't name a sibling alternative.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• runWithSecretsA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden and discloses key behaviors: redacted output, exit code, and that secret values are never exposed to the model. This is critical security context. It does not cover error handling or environment details, but the core transparency is strong.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three concise sentences with no redundancy: purpose, output, and security guarantee. Each sentence adds distinct value, and the core action is front-loaded. This is efficient and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no output schema and no annotations, the description covers the essential behavior and safety promise. It omits details like failure modes or how secrets map to environment variable names, but an agent has enough to select and invoke it correctly. Slightly above average due to the security-focused completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate. It adds meaning for 'secretNames' by explaining they are injected as environment variables, and 'command' is self-evident. However, it does not mention 'args' at all, leaving that parameter undocumented. The description partially compensates but not fully.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('run'), clearly identifies the resource ('command with named secrets'), and distinguishes itself from sibling secret-management tools by focusing on execution with injected secrets. It also states the output behavior (exit code, redacted output), making the purpose unmistakable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage: use this when you need to run a command with secrets without exposing them. It does not explicitly name alternatives or when-not-to-use, but the security guarantee ('never exposed to the model') suggests it is the safe choice over manually retrieving secrets. This is clear context without explicit exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.