Server Quality Checklist

Tool Scores

• set_secretA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It only states 'Creates or overwrites' but does not disclose whether overwriting is destructive, authentication requirements, rate limits, or error behavior. Minimal behavioral insight.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two short sentences convey the core action and a key requirement. No redundant or superfluous information. Excellent front-loading of purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple tool (2 params, no output schema, no annotations), the description should be more complete. Missing details: what happens if name is not whitelisted, overwrite behavior, return value, error handling, or data sensitivity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0% with no parameter descriptions in the schema. The description only adds context for 'name' (must be whitelisted) but provides no meaning for 'value' beyond being the secret. Value lacks type, constraints, or formatting guidance.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action: 'Store a secret value' and mentions 'Creates or overwrites.' It distinguishes from siblings like delete_secret, get_secret, list_secrets, etc., by focusing on storage.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides a prerequisite: 'Requires name to be whitelisted in project .claude/secrets.yml.' This gives clear context for when to use the tool, though it does not explicitly exclude alternative tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• delete_secretA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description reveals deletion behavior and permission requirement, but lacks details on side effects (e.g., irreversibility, cascading deletes) or error handling.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that conveys the essential action and a key prerequisite. No superfluous words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple delete tool with one parameter and no output schema, the description covers the basic action and a requirement. However, it lacks details on idempotency or handling of missing secrets, leaving some context gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description says 'by name', which implies the parameter is a name, but does not elaborate on format or constraints. Schema has no description, so the description adds minimal value beyond identifying it as the identifier.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'delete' and the resource 'secret by name', which distinguishes it from sibling tools like get_secret, list_secrets, etc. The purpose is immediately understandable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It mentions 'Requires allowlist', giving a precondition, but does not specify when to use this tool vs alternatives (e.g., set_secret). No guidance on when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_secretsA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It discloses case-insensitive regex and allowlist filtering, but does not specify return format (names vs. full secrets), behavior on no match, or error handling.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence front-loading the action and key constraints (regex, case insensitive, allowlist filter). No extraneous words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-param tool with no output schema and no annotations, the description is mostly adequate. However, it lacks details on return values and error behavior, which would be helpful for an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0% (no description for 'pattern'). Description adds that it is a regex pattern and case insensitive, which provides essential semantic meaning beyond the schema's raw type.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the verb 'Search', resource 'secret names', and method 'by regex pattern (case insensitive)'. It distinguishes from siblings like 'list_secrets' by specifying a regex search, and 'get_secret' by targeting names only.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Implied usage: use when needing to find secret names matching a pattern. No explicit mention of when not to use or alternatives among siblings like 'list_secrets' or 'get_secret'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_secretA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses the authorization requirement (whitelisting), which is a behavioral trait. However, it does not mention side effects (none expected), rate limits, or response format, leaving some gaps for a complete picture.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no fluff. It front-loads the action and then adds the prerequisite. Every sentence is necessary and efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the low complexity (1 parameter, no output schema), the description covers the core action and a critical prerequisite. However, it omits what the tool returns (e.g., the secret value in plaintext), which would be helpful for an agent without an output schema. Still, it is fairly complete for a simple read operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% for the only parameter 'name', so the baseline is 3. The description does not add any additional meaning beyond the schema's 'Secret name' description. No extra syntax, format, or constraints are provided.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Read a secret value by name', which is a specific verb+resource combination. It distinguishes from sibling tools like delete_secret, list_secrets, etc., which have different actions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides a prerequisite (whitelisting) but does not explicitly state when to use this tool versus alternatives. Usage is implied by the action 'read' compared to siblings, but no when-not-to-use or explicit alternative naming is given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_secretsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. Mentions filtering by allowlist, but lacks details on read nature, auth requirements, or result format. Adequate but not fully transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, efficient, front-loaded with key information. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Tool is simple (no parameters, no output schema). Description covers purpose and filtering context. Could mention whether pagination exists, but not critical for a names-only list.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters, so schema coverage is 100%. Baseline for zero parameters is 4. Description adds no parameter info, but none needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states it lists secret names, with filtering by an allowlist. Verb 'list' with resource 'secret names' is specific. Distinguishes from siblings like delete_secret or set_secret by focusing on names only.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Description tells what it does but not when to use it over siblings like search_secrets or get_secret. No explicit when-not or alternatives, though sibling names hint at different actions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• input_secretA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Describes key behavioral trait: value never passes through model or chat, goes directly to encrypted store. No annotations provided, so description carries full burden. Could mention whether it overwrites existing secrets or returns anything.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with purpose. Every sentence adds essential information. No waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  No output schema, but description gives sufficient context for a simple interaction tool. Could clarify that dialog blocks execution, but overall complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, baseline 3. Description adds value by noting name must be whitelisted in secrets.yml, which is not in schema. Also explains prompt parameter context.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states it prompts user for a secret via macOS dialog and stores it, using specific verb 'Prompt' and resource 'secret'. Distinguishes from siblings like get_secret, set_secret by emphasizing user interaction.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit usage guidance: 'Use this when you need a token, password, or API key the user has not yet stored.' Also mentions prerequisite (name whitelisted). Could add explicit when-not-to-use, but inferred.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.