Server Quality Checklist

Tool Scores

• read_fileC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description must carry behavioral disclosure. It mentions RBAC enforcement but lacks details on error handling (e.g., file not found), encoding, behavior for binary files, or truncation with max_chars. The term 'text-like' is vague.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single short sentence, but it omits essential details. Conciseness should not come at the cost of completeness; under-specification reduces utility.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations, incomplete parameter documentation, and no output schema info, the description does not adequately cover the tool's usage, behavior, or return values. Siblings exist but no guidance.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description should compensate. It does not explain the meaning of rel_path (that it's relative to repo root) or max_chars (truncation limit). The path scope is hinted but not explicitly tied to the parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Read'), the resource ('text-like file'), and the scope ('relative to repo root'). It distinguishes from siblings: list_files lists files, search searches, set_role sets role, so the purpose is unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for reading file contents but does not explicitly state when to use it vs. siblings like list_files or search. No guidance on when not to use or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• set_roleB
  3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must fully disclose behavior. It mentions that domains are returned, but does not explain side effects like overriding previous role, reversibility, or impact on other operations. This is insufficient for a state-modifying tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no filler. The description is front-loaded and every word adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple with one parameter and an output schema (implied by context). The description covers the return value, but lacks details about parameter values and any usage context. It is minimally adequate for a straightforward tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, meaning the description must compensate. However, the description does not mention the 'role' parameter at all, nor its enum values or purpose. It only says 'Set the active role' without linking to the parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states that the tool sets the active role for the MCP session and returns accessible domains. It uses a specific verb and resource, and distinguishes from sibling tools like list_files, read_file, and search.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus alternatives. The description implies it is for changing the session role, but does not mention prerequisites, exclusions, or when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• searchA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It mentions RBAC awareness and lists return fields, but lacks disclosure of side effects, rate limits, or authentication requirements.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is structured with a clear first sentence and a parameter list, but it is somewhat verbose and repeats information that could be inferred from the schema, though it remains efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (3 parameters, output schema exists), the description adequately covers purpose, parameters, and return format. It misses prerequisites or limitations but is largely complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, but the docstring adds meaningful parameter explanations: 'natural language query', 'number of results to return', and 'include general alongside role-specific domains', which go beyond plain property names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it is a vector search tool with RBAC awareness, specifying it searches across allowed domains/collections. It distinguishes itself from sibling tools like list_files and read_file by focusing on search.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for search with role-based access but does not provide explicit when-to-use or when-not-to-use guidance, nor does it compare to alternatives like list_files or read_file.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_filesA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses RBAC enforcement (access control) and the return format (relative paths from repo root). No annotations are provided, so the description carries the full burden, and it adds meaningful behavioral context beyond the schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, zero wasted words. The first sentence captures the core action and access model, the second explains the output. Information is front-loaded and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the low complexity (1 required parameter with enum, output schema exists) and sibling tools, the description covers the essential behavior: what it lists, where, and what returns. Minor gaps like recursion depth or sorting are not critical for this tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description must compensate. It explains that the 'domain' parameter corresponds to a subdirectory under 'data/', which adds meaning beyond the enum list in the schema. This helps the agent understand the parameter's role.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'List files under data/<domain>' which is a specific verb and resource. It distinguishes from sibling tools like read_file (reads a single file) and search (queries content) by focusing on listing files within a domain directory.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus alternatives. It does not state prerequisites, when not to use, or mention any filtering capabilities. The description implies usage for listing files in a domain but lacks context for decision-making.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.