Server Quality Checklist

Tool Scores

• vault_list_directoryA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, openWorldHint, idempotentHint, and destructiveHint=false. The description adds value beyond that by disclosing error messages ('Error: ... not found'), rate-limit behavior and the GITHUB_TOKEN tip, and the exact JSON return structure. While it repeats some annotation info, it contributes meaningful behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is organized into clear sections (Args, Returns, Examples, Error Handling) with a front-loaded purpose statement. Every sentence contributes: examples clarify usage, the 'Don't use when' routes to a sibling, and error handling preempts common failures. No fluff or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a list-directory tool with no output schema, the description provides everything an agent needs: the return format (JSON), error handling, rate-limit advice, and usage examples. It also differentiates from siblings. Open-world concerns like large results are partly addressed by the rate-limit note. Nothing critical is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so parameters are fully documented in the schema (defaults, descriptions, enum). The description repeats this information and adds usage examples ('governance' path) but does not introduce new semantic meaning not already in the schema. Baseline 3 is appropriate; the examples are marginal added value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action and resource: 'List the files and subdirectories at a given path in the canonical-vault repository'. It explicitly names the sibling tool it is not (vault_get_file) and provides usage examples that clarify the exact scope. This clearly distinguishes it from the other siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives explicit when-to-use examples ('What's in the governance directory?') and an explicit when-not-to-use condition ('Don't use when: You need the actual contents of a file (use vault_get_file instead)'). This is the clearest possible guidance, leaving nothing to inference.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• vault_get_fileA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, openWorldHint=true, idempotentHint=true, and destructiveHint=false. The description adds valuable behavioral detail beyond these: it explains the returned format (header with path, ref, sha, size), imposes a size limit (>500000 bytes returns error), and specifies error messages for missing files and directories. This goes beyond the annotation baseline, though it doesn't describe potential rate limits or exhaustive edge cases, which is acceptable for a read-only file fetch.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is longer than average but highly structured: a one-sentence purpose, then separate 'Args', 'Returns', 'Examples', and 'Error Handling' sections. The most critical info (purpose and primary use case) is front-loaded, and every section adds distinct value. It is not rambling; it is comprehensive yet organized. Minor redundancy exists between the error handling and the 'Don't use when' note, but not enough to lower the score further.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Without an output schema, the description fully explains the return format (raw text with a small header) and error cases (not found, directory, binary, oversized). It covers both required and optional parameters, provides examples, and names the relevant sibling tools. For a tool of this complexity, nothing essential is missing—an agent can call it correctly and interpret results without additional information.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with descriptions for both 'path' and 'ref'. The description adds meaning beyond the schema by providing an example for path ('governance/CONSTITUTION.md'), clarifying that paths are relative to repo root, and listing allowed ref values. This enriches the baseline of 3 for full schema coverage, making the parameters more intuitively usable.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb-resource pair: 'Fetch the decoded text content of a single file from the canonical-vault repository'. It clearly distinguishes from siblings like vault_list_directory (which lists) and vault_search_files (which searches), and the scope is explicitly limited to a single file. This leaves no ambiguity about the operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit when-to-use and when-not-to-use guidance. It provides concrete natural-language examples (e.g., 'Read me the current Constitution' → path='governance/CONSTITUTION.md') and directly tells the agent to use vault_list_directory for existence/size checks. It also covers the case of directories and binary files, routing them to the sibling. This is exemplary.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• vault_get_commitA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare the operation as read-only, idempotent, and non-destructive. The description adds valuable behavioral details beyond those: the exact JSON return structure, the default response format, and the error handling when a SHA is not found. This goes beyond simple scoping and genuinely helps an agent anticipate tool behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (Args, Returns, Examples, Error Handling) and front-loads the core purpose. Every sentence earns its place—no fluff. It is longer than a single sentence, but the length is justified by the need to document output format and error behavior given the absence of an output schema.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no output schema, the description fully documents the return format (both markdown and JSON structures), error handling, and usage examples. It also clarifies the scope ('in canonical-vault') and sources the sha from a sibling tool. An agent has everything needed to invoke it correctly, and nothing is left to inference.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the schema already documents both parameters. The description adds extra meaning by providing a concrete example of how to obtain a sha (from vault_get_file_history) and by illustrating the response_format usage. While not essential, this context adds value beyond the schema's bare parameter definitions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a clear, specific verb-resource statement: 'Get full detail for a single commit in canonical-vault: message, author, stats, and the list of files it changed (with per-file diff stats).' This precisely defines the tool's function and distinguishes it from siblings like vault_get_file_history (which returns commit lists) and vault_get_file (which retrieves file content).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit usage guidance is provided: a 'Use when' example ('What exactly changed in commit a1b2c3d?') and a 'Don't use when' case that names the alternative tool (vault_get_file_history). This leaves no ambiguity about when to select this tool over its siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• vault_get_file_historyA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate readOnly, idempotent, and non-destructive behavior, but the description adds substantial behavioral detail: it returns a list of commits with fields, supports pagination via offset, explains response_format options, and lists error handling ('Error: ... not found'). This goes well beyond the annotations by describing the exact behavior and output shape.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (Args, Returns, Examples, Error Handling) and is front-loaded with the purpose. While it is fairly long, every section serves a purpose: documenting parameters, return format, and usage guidance. It is not bloated, but could be trimmed slightly without losing value, hence 4 rather than 5.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given there is no output schema, the description compensates by specifying the exact JSON return structure. It covers all 5 parameters, provides usage examples, error cases, and pagination semantics. Nothing an agent needs to call this tool correctly is missing; it is complete for its complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema covers 100% of parameters with descriptions, so baseline is 3. The description adds marginal value by clarifying parameter usage, e.g., that path='' means whole-repo history, offset is for pagination, and response_format controls output. It also gives concrete examples that illuminate parameter semantics. This nudges the score slightly above baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description states a specific action ('Get the commit lineage') on a specific resource ('file or directory in canonical-vault') and immediately clarifies it returns every commit that touched that path, most recent first. It also explicitly differentiates from sibling tools, e.g., 'Don't use when... use vault_get_commit instead.' This makes its purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear when-to-use examples ('How has VARA_SPEC_1.0 evolved?') and explicitly states when not to use it, referencing an alternative tool (vault_get_commit). It also notes default behavior for whole-repo history. This gives the agent actionable guidance on selecting this tool over siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• vault_search_filesA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the annotations (readOnlyHint, idempotentHint, etc.), the description discloses meaningful behavioral traits: GitHub code search limitations (only default branch, size limits, indexing lag), the exact error message when no results are found, and the return structure for JSON format. No contradictions with annotations exist.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (Args, Returns, Examples, Error Handling) and is front-loaded with the core purpose. It repeats some schema details (like parameter defaults) but organizes them efficiently, and every sentence contributes useful information without being verbose or redundant.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is complete for a search tool with pagination, multiple output formats, and external dependencies. It covers search syntax, pagination (offset), format handling, error messages, and known limitations. Since there is no output schema, it explicitly defines the JSON return structure, ensuring the agent knows exactly what to expect.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the schema already documents each parameter. However, the description adds value beyond the schema by supplying practical examples (e.g., 'path:governance' qualifier), clarifying pagination behavior via offset, and specifying what each response_format yields. This extends the schema's dry definitions with actionable context.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Search'), a clear resource ('file names and contents across the canonical-vault repository'), and the method ('using GitHub code search'). This distinguishes it from sibling tools like vault_list_directory or vault_get_file, which serve different actions. The purpose is unambiguous and directly informs the agent when to invoke this tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit 'Use when' and 'Don't use when' examples, including the exact condition for selecting an alternative (already knowing the exact path → use vault_get_file). It also gives concrete query examples and notes error handling, giving the agent clear decision criteria for when to apply this tool versus its siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.