Server Quality Checklist

Tool Scores

• rule_findingsA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It details multiple behavioral aspects of the tool: it surfaces rules that are ignored, misanchored, over-broad, or shadowed, and each finding includes documented behavior. It does not explicitly state that it is read-only, but the analytical nature is evident. This is above minimum but could say more about outputs.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences and packed with useful detail, but the first sentence is quite long and lists many edge cases. It is still concise and every part contributes to understanding the tool's purpose, though a more front-loaded verb would improve scannability.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has no output schema, so the description should clarify what the results look like. It only says 'Each finding cites the documented behaviour,' which gives a hint but not the complete shape of the output. Given the tool's diagnostic nature and the rich input schema, the description is adequate but leaves room for more detail.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already has full descriptions for both 'dir' (absolute path, default working directory) and 'kind' (enum with five categories). The description does not add any additional parameter-specific details, so it relies entirely on the schema. Baseline of 3 applies since schema coverage is 100%.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly defines the tool's scope: it identifies permission rules whose documented behavior differs from their apparent intent. It enumerates specific categories (inert, misreads, wider, shadowed) which makes the purpose concrete, but it lacks an explicit verb like 'find' or 'list,' so it's slightly less direct than a 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The final sentence gives concrete guidance: 'Use before trusting a guard rail you wrote a while ago.' This indicates when to use the tool, but it does not explicitly mention alternative sibling tools or when not to use it. Still, the context is clear and actionable.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• unattended_surfaceA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden. It discloses the output content (defaultMode, allow rules, directories, bypass/auto disabled, hook commands) and notes hook commands run without a permission prompt. It does not explicitly state side-effect-free or read-only, but the nature of the config inspection implies safety. This adds meaningful behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the tool's core purpose, followed by usage guidance. No wasted words, easy to scan.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the single optional parameter and no output schema, the description provides a good list of what the result contains. It also ties into the intended scenario (unattended agents) and repository review, making it sufficiently complete for the tool's simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description covers the single parameter 'dir' with a clear explanation. The tool description does not add any parameter-specific detail beyond that, but with 100% schema coverage, the baseline of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: exposing configuration elements that allow tool calls to proceed without human supervision, listing specific components. This distinguishes it from sibling tools like whats_allowed or permission_sources by focusing on the 'unattended' operation context.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit usage contexts: 'Use before leaving an agent running unattended, or when reviewing what a repo's settings would do on your machine.' However, it does not explicitly contrast with alternatives, so it stops short of full when/when-not guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• whats_allowedA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations present, the description takes on the full burden of behavioral context. It discloses that the tool returns a summary containing specific details, including a note about rules that do not behave as they appear. It does not explicitly state that the tool is read-only, but the nature of a summary implies no side effects. This is adequate transparency for a simple informational tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise yet information-dense, with two sentences. The first sentence front-loads the tool's purpose and lists its output contents in a structured list, while the second provides precise usage guidance. Every phrase contributes to understanding, with no redundant or vague language.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with one optional parameter and no output schema, the description sufficiently explains what the tool returns and when to use it. It covers the key aspects of the tool's functionality, though it does not explicitly describe the output format. However, the list of contents is enough for an agent to understand the tool's role within the sibling tool set.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 100% description coverage for the single parameter 'dir', including a default. The tool description also mentions 'directory', reinforcing the schema. Since the schema already documents the parameter meaning, the description adds no additional semantic value beyond what is present in the schema, so the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the tool as a one-call summary of the permission configuration for a directory, enumerating specific elements such as contributing settings files, allow/ask/deny rule counts, defaultMode, blanket allows, hooks, and rule mismatches. This distinguishes it from siblings like permission_sources and rule_findings, which are likely more detailed or narrow in scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly states when to use the tool: at session start, before an unattended run, or when an unexpected prompt appears. This provides clear context, but it does not mention when not to use it or explicitly name alternatives, though 'Start here' implies it is the entry point to the permission investigation process.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• permission_sourcesA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses what the tool reports (existence, parsed status, rule count, path anchor meaning) and that it lists files in precedence order. It doesn't explicitly state read-only, but the informational nature is evident. It adds useful behavioral context beyond a generic list.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences: the first defines the tool's output and scope, the second gives usage guidance. No fluff, front-loaded with core functionality, and every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite the absence of an output schema, the description fully enumerates what the tool returns (existence, parse status, rule count, path anchor meaning) and covers precedence order and use cases. It is complete for a single-parameter listing tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% coverage for the only parameter 'dir', including its meaning and default. The tool description adds no additional parameter detail, so the baseline of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the tool as enumerating every settings file that can contribute permission rules, in documented precedence order, with specific details per file (existence, parse status, rule count, path anchor semantics). It distinguishes itself from siblings by stating use cases like 'when a rule is not taking effect' or 'when you cannot tell which file granted something'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage scenarios: 'Use when a rule is not taking effect, when you cannot tell which file granted something, or to confirm a managed policy is or is not present.' It lacks named alternatives or explicit 'when not to use', but the guidance is clear and context-rich.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.