Server Quality Checklist

Tool Scores

• get_secretA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavior. It only states the retrieval action without confirming read-only nature, error handling, output format, or security context. This leaves the agent to infer safety, which is a significant gap for a secret retrieval tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single sentence that is front-loaded and contains no waste. It efficiently communicates the core purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter retrieval tool, the combination of description and schema covers the essentials, but the lack of annotations, output schema, and behavioral details (e.g., errors, return format) leaves moderate gaps. It's functional but not fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema fully describes the path parameter (100% coverage), and the description's 'by path' reinforces this without adding new semantics. The baseline of 3 applies because schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'Retrieve' with the resource 'secret's content' and scope 'from gopass by path,' clearly distinguishing it from siblings like list_secrets (listing) and generate_password (creating). The intent is unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when you know the secret's path ('by path'), but it does not explicitly state when to use this over list_secrets or search_secrets, nor does it mention exclusions. The guidance is only implicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_secretsA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description must disclose all behavioral traits. It mentions the optional filter and that it lists paths (not contents), but does not state if the operation is read-only (implied), how results are ordered or truncated, or whether it works with nested prefix hierarchies. The behavioral disclosure is minimal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, front-loaded with the action and resource, and includes the optional filter without any extraneous words. No wasted content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool is simple (1 optional parameter, no output schema, no nested objects), the description adequately covers the main purpose. However, it lacks details on return format (flat list vs. tree), error states (e.g., no such prefix), and performance considerations (e.g., large stores). It is minimally complete but not comprehensive.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the parameter is already documented in the schema. The description adds the context that the prefix is a 'folder prefix' rather than a flat tag, which adds slight value over the schema's description. Baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb ('List') and resource ('secret paths in the gopass store'), and includes the optional filtering capability ('optionally filtered by folder prefix'), which distinguishes it from siblings like get_secret and search_secrets.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for listing secrets, but does not explicitly say when to use this versus search_secrets (which might filter by content) or get_secret (which retrieves a single secret). No when-not-to-use or alternative guidance is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_secretsA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full responsibility for behavioral disclosure. It only states that the tool searches by name, omitting critical details: matching semantics (exact, prefix, fuzzy), case sensitivity, return format, number of results, behavior on no matches, or whether it lists matches or content. This is insufficient for an AI agent to predict the tool's behavior confidently.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with no filler. Every word is essential and front-loaded with the verb and resource. It is appropriately sized for the tool's simplicity and avoids any redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (one parameter, no output schema or annotations), the description is minimally adequate: it states the core function. However, it lacks information about return values, error handling, and search behavior details that would make it complete. An agent would need to infer or discover these through interaction.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema covers 100% of parameters, but the parameter description ('Search query') is generic. The tool description adds meaning by specifying that the query is matched against secret names, which the schema alone does not convey. This extra context helps the agent understand what value to provide in the 'query' parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's action ('Search') and resource ('secrets by name in the gopass store'). It effectively distinguishes from siblings: list_secrets (listing all), get_secret (retrieving a specific secret), and generate_password (creating new passwords). The verb-resource pairing is specific and unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use search over listing all secrets (list_secrets) or retrieving a specific secret (get_secret). There is no mention of prerequisites, typical use cases, or situations where this tool is preferred. The agent is left to infer usage from the tool name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• generate_passwordA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description bears full responsibility for behavioral disclosure. It correctly indicates the tool creates and stores a random password, but it fails to mention what happens if the path already exists (overwrite, error, or append) or whether any special permissions are required. This is a notable gap for a mutation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence of 13 words with no filler. Every word contributes to understanding the tool's core action and context.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of an output schema, the description should indicate what the tool returns (e.g., the generated password or a confirmation). It does not, which is a gap. However, the parameter documentation is complete, and the core behavior is explained, making it adequate but not comprehensive.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and both parameters have descriptive comments in the schema. The description adds the term 'random' and 'gopass' but does not provide meaningful additional semantics beyond what the schema already offers, so a baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('generate and store a random password'), the resource ('at the given path'), and the system ('gopass'). This distinguishes it from sibling tools which are read operations (list, get, search).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The purpose is clear, and sibling tools are all read-based, so an agent can infer this is for creating new password entries. However, there is no explicit guidance on when to avoid this tool (e.g., if the path already exists) or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.