Server Quality Checklist

Tool Scores

• delete_sectionC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool deletes a section, implying a destructive mutation, but doesn't specify whether this action is reversible, requires specific permissions, has side effects (e.g., on associated secrets), or provides confirmation. This leaves significant gaps in understanding the tool's behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's purpose without any unnecessary words. It is front-loaded and wastes no space, making it highly concise and well-structured for quick understanding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's destructive nature (deleting a section), lack of annotations, and no output schema, the description is insufficient. It doesn't cover critical aspects like what happens on success or failure, error conditions, or the impact on related data (e.g., secrets within the section). For a mutation tool with zero annotation coverage, this is a notable gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, with the 'title' parameter clearly documented. The description adds no additional semantic context beyond what the schema provides, such as format examples or constraints. According to the rules, with high schema coverage, the baseline is 3, which is appropriate here.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Delete') and resource ('a section from the secrets vault by title'), making the purpose unambiguous. However, it doesn't explicitly differentiate from sibling tools like 'upsert_section' which might also modify sections, so it doesn't reach the highest score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'upsert_section' for updates or 'get_section' for retrieval. It lacks context about prerequisites, such as whether the section must exist or be empty, and doesn't mention any exclusions or recommended alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_sectionC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden but only states the basic action. It doesn't disclose behavioral traits like error handling (e.g., if section doesn't exist), authentication requirements, rate limits, or return format. For a read operation in a secrets vault, this leaves critical gaps in understanding how it behaves.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero waste. It's front-loaded with the core purpose, making it easy to parse quickly. Every word earns its place without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description is incomplete for a tool accessing a secrets vault. It lacks details on security implications, response structure, or error cases. For a sensitive operation like retrieving secrets, more context is needed to ensure safe and correct usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with the parameter 'title' fully documented in the schema. The description adds minimal value by repeating 'by title' but doesn't provide additional context like format constraints or examples. Baseline 3 is appropriate as the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Get') and resource ('a specific section from the secrets vault'), specifying it retrieves by title. It distinguishes from siblings like 'list_sections' (all sections) and 'search_secrets' (broader search), but doesn't explicitly contrast with 'delete_section' or 'upsert_section'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives is provided. It doesn't mention prerequisites (e.g., authentication), differentiate from 'search_secrets' for partial matches, or explain when to prefer 'list_sections' for browsing. The description implies usage for exact title matches but lacks explicit context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_secretsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It mentions 'Search' but doesn't clarify if this is a read-only operation, whether it requires authentication, what happens with no matches, or how results are returned. For a search tool with zero annotation coverage, this leaves significant behavioral questions unanswered.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that gets straight to the point without unnecessary words. It could be slightly improved by front-loading more critical information about behavioral aspects, but it's appropriately sized for a simple search tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a search tool with no annotations and no output schema, the description is inadequate. It doesn't explain what 'sections' are, what format results are returned in, whether there are limitations on search scope, or how this differs from other retrieval tools. The description should provide more context given the lack of structured metadata.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% with the single parameter 'keyword' well-documented in the schema. The description adds minimal value beyond the schema by mentioning 'matching a keyword' but doesn't provide additional context about search semantics, case sensitivity, partial matching, or search scope. Baseline 3 is appropriate when schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Search') and target resource ('secrets vault for sections'), making the purpose understandable. However, it doesn't differentiate from sibling tools like 'list_sections' or 'get_section' - it mentions 'matching a keyword' but doesn't explain how this differs from other listing/retrieval operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided about when to use this tool versus alternatives like 'list_sections' or 'get_section'. The description implies keyword-based searching but doesn't specify scenarios where this is preferable to other retrieval methods or mention any prerequisites for using the tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• upsert_sectionC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It implies a mutation operation ('Add or update') but fails to detail critical aspects like whether this requires specific permissions, if updates are idempotent, what happens on conflicts, or any rate limits. This leaves significant gaps in understanding the tool's behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's purpose without any fluff or redundancy. It is appropriately sized and front-loaded, making it easy for an AI agent to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of a mutation tool with no annotations and no output schema, the description is insufficient. It lacks details on behavioral traits, error handling, return values, and usage context, making it incomplete for safe and effective tool invocation by an AI agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, so the input schema already documents all parameters (title, content, level) with descriptions and constraints. The description adds no additional meaning beyond what the schema provides, such as explaining how 'title' and 'content' interact or the implications of 'level'. Baseline 3 is appropriate as the schema handles the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Add or update') and resource ('a section in the secrets vault'), making the purpose specific and understandable. However, it doesn't explicitly differentiate from sibling tools like 'delete_section' or 'get_section' beyond the implied action difference, which keeps it from a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives such as 'delete_section' or 'get_section', nor does it mention prerequisites like authentication or context for updates versus additions. This lack of explicit usage instructions limits its helpfulness for an AI agent.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_sectionsB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It states the action but doesn't disclose behavioral traits such as whether this is a read-only operation, if it requires authentication, rate limits, or what the return format looks like (e.g., list structure, pagination). This leaves significant gaps for agent understanding.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence with zero waste. It's front-loaded with the core action and resource, making it highly efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations and output schema, the description is incomplete. It doesn't explain what 'list all section titles' entails—such as the return format, ordering, or any limitations. For a tool with no structured behavioral data, this leaves the agent with insufficient context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters, and schema description coverage is 100%, so there are no parameters to document. The description appropriately doesn't add parameter details, earning a high baseline score for not introducing unnecessary information.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('List') and resource ('all section titles in the secrets vault'), making the purpose immediately understandable. However, it doesn't explicitly differentiate from sibling tools like 'get_section' or 'search_secrets', which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'get_section' (for a specific section) or 'search_secrets' (for searching within sections). It lacks context about prerequisites, timing, or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.