Server Quality Checklist

Tool Scores

• vet_command_chainA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must fully disclose behavior. It explains severity escalation (LOW→MEDIUM, MEDIUM→HIGH) and cites a real incident, but it does not specify what the vetting result is (e.g., risk score, flag, block) or how to interpret the output. This leaves some ambiguity.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is about 4 sentences and relatively concise. The anecdote about r/LocalLLaMA adds context but could be trimmed. Front-loads the core purpose. Slightly verbose but acceptable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given one required parameter, no output schema, and no annotations, the description adequately covers usage context and behavior difference from sibling. However, it lacks details on response format, error cases, or what success/failure looks like. For a simple tool, this might be sufficient, but more completeness would help.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema description coverage, baseline is 3. The description adds nuance by clarifying what 'chained' means (&&, ||, ;, piped subshells), but this largely echoes the schema's description of 'The chained shell command to vet' without adding significant new meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool vets chained shell commands and escalates severity compared to vet_command. It provides specific examples of chain operators (&&, ||, ;, piped subshells) and references a real-world failure mode, effectively distinguishing it from its sibling.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use: 'Use this for any command containing &&, ||, ;, or piped subshells.' The description also explains the rationale (operator oversight). It does not explicitly state when not to use, but the sibling name implies single commands should go to vet_command.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_detection_rulesA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Since no annotations are provided, the description carries full burden. It transparently states the operation is a catalog retrieval (read-only) and provides details on contents (30 rules, 8 families). No side effects are implied, which is appropriate for a list tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences: first states purpose and output, second gives use cases and summary. No fluff, front-loaded with essential information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite no output schema or annotations, the description fully covers what the tool returns (fields, families, count). An agent can confidently invoke and interpret results.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has no parameters, so schema coverage is 100%. The description adds value by explaining the output details, meeting the baseline for a zero-parameter tool.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states the tool returns a catalog of every detection rule with specific fields (rule_id, severity, pattern_kind, description, example_match). It clearly distinguishes from siblings (vet_command, vet_command_chain) which are likely for vetting, not listing.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear use cases: audit coverage, document detection scope, build custom allowlist. It implicitly suggests this is the go-to tool for listing rules, with no mention of alternatives for exclusion.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• vet_commandA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description fully discloses the tool's behavior: it detects various destructive patterns, returns a verdict with risk_score and per-finding details, and operates sub-second locally with no API key. There are no annotations, so the description carries the full burden, which it meets.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the main purpose, followed by an enumeration of detections and return fields. Every sentence adds value, and there is no redundant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the single parameter and no output schema, the description is complete: it explains what the tool does, what it detects, what it returns, and its performance characteristics (sub-second, local, no API key).

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% for the single 'command' parameter, so baseline is 3. The description does not add significant parameter-specific semantics beyond what the schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: vetting a single shell command for destructive patterns before execution. It lists specific patterns detected, distinguishing it from sibling tools like vet_command_chain.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides clear context: 'Use inline before approving any agent-proposed command' and mentions performance characteristics. However, it does not explicitly contrast with the sibling tool vet_command_chain for when to use this vs. the chain variant.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.