Server Quality Checklist

Tool Scores

• arai_list_guardsA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Describes a read-only operation (list) returning triples and source files. No annotations provided, so description handles disclosure adequately. Missing details like potential performance for large lists or authorization.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no wasted words. Front-loaded with purpose, followed by return value and usage hint.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Complete for a simple list tool with one optional parameter and no output schema. Covers purpose, filtering, return format, and usage timing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Parameter 'pattern' is fully described in schema (case-insensitive substring). Description adds no new semantic info beyond schema, achieving baseline 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states it lists guardrails with optional filtering. Distinguishes from sibling tools (add, check, recent) by focusing on listing active constraints.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly mentions using it 'before making a tool call' to see live constraints. Does not explicitly state when not to use or give alternatives, but context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• arai_add_guardA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It discloses that the rule is parsed like CLAUDE.md instructions, stored locally, and takes effect on the next PreToolUse hook, providing important behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, front-loaded with the core purpose, and every sentence adds necessary information without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple 2-param tool with no output schema, the description covers purpose, usage, and behavior adequately. It could mention error handling or limits, but overall it is complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with descriptions, but the description adds value by giving examples for 'rule' and explaining that 'reason' is for audit log, enhancing understanding beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool registers a new guardrail for enforcement, using a specific verb ('register') and resource ('guardrail'). It distinguishes itself from siblings like arai_list_guards (listing) and arai_check_action (checking).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Use when you discover a rule mid-session that should persist' and provides examples, giving clear context. It does not explicitly state when not to use, but the purpose is well-defined.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• arai_check_actionA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. Discloses no execution, no audit log, returns matched rules with severity and source, same shape as arai_recent_decisions. Could add more on side effects but sufficient for a probe-only tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences: first states purpose and key behavior, second gives usage guidance and return shape. Every word earns its place, no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 3-param tool with nested objects and no output schema, the description covers purpose, when to use, input examples, and return format. Sufficient for an agent to use correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has 100% description coverage. Description adds value by explaining the shape of tool_input for different tools (Bash, Edit/Write) and describing the return shape, going beyond schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses specific verb 'Probe' and resource 'guardrail matching', clearly distinguishing from siblings: arai_add_guard adds, arai_list_guards lists, arai_recent_decisions returns actual firings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states 'Use BEFORE taking an action you think might be regulated to avoid a deny-and-retry loop' and clarifies it doesn't execute or write audit log, implying when not to use. Lacks explicit naming of alternatives but context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• arai_recent_decisionsA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses that the tool returns tool, decision type, matched rules, and source file. It also explains the default time window and session scoping. However, it does not explicitly state whether the operation is read-only or if there are side effects, though the nature of the tool suggests it is safe.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences with no extraneous content. It front-loads the purpose, then gives usage guidance, and finally details the return value. Every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 3 optional parameters and no output schema, the description adequately explains the purpose, usage, and return fields. It covers default behavior for parameters and mentions the return structure. It could be improved by noting result ordering (implied as most recent) or error handling, but it's still reasonably complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, giving a baseline of 3. The description adds meaning beyond the schema by explaining that session_id should match Claude Code hook payloads and that since defaults to 24 hours to avoid stale entries. This provides useful context not present in the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool looks up recent guardrail decisions, using the verb 'look up' and specifying the resource as 'guardrail decisions'. It differentiates from sibling tools like arai_add_guard (adding guards) and arai_list_guards (listing guards) by focusing on decisions, not guards themselves.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly advises when to use the tool: 'Use this when you've just been denied or warned and want to check whether you've hit the same rule before.' It provides a concrete use case and explains the benefit of closing the feedback loop.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.