Server Quality Checklist

Tool Scores

• check_actionC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool checks action permissions but lacks details on what 'covenant rules' entail, whether this is a read-only operation, if it has side effects, error handling, or performance characteristics. For a tool with zero annotation coverage, this is insufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence that efficiently conveys the core purpose without unnecessary words. It's front-loaded with the main function and appropriately sized for the tool's complexity, making it easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations and output schema, the description is incomplete. It doesn't explain what the tool returns (e.g., a boolean, detailed rule match, error messages) or behavioral aspects like idempotency or rate limits. For a permission-checking tool with no structured context, more detail is needed to guide effective use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, so the input schema already documents both parameters ('action' and 'params') thoroughly. The description adds no additional meaning beyond what the schema provides, such as examples of 'covenant rules' or how parameters interact with them. Baseline 3 is appropriate when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Check whether an action is allowed or blocked by the current covenant rules.' It specifies the verb ('check') and the resource/scope ('action' against 'covenant rules'), making it easy to understand what the tool does. However, it doesn't explicitly differentiate from sibling tools like 'verify_log' or 'set_rules', which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention sibling tools like 'get_audit_log', 'set_rules', or 'verify_log', nor does it explain prerequisites, typical use cases, or exclusions. This leaves the agent without context for tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_audit_logB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions 'full hash-chained audit trail', hinting at data structure and completeness, but fails to address critical aspects like whether this is a read-only operation, potential performance impacts, rate limits, or authentication requirements for a compliance-related tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose ('Returns the full hash-chained audit trail') without any fluff. Every word earns its place, making it highly concise and well-structured for quick comprehension.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of audit trails and compliance checks, the description is insufficient. With no annotations and no output schema, it lacks details on return format (e.g., structure of the hash chain), data volume, or error handling. This leaves significant gaps for an agent to use the tool effectively in a compliance context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0 parameters with 100% coverage, so no parameter documentation is needed. The description appropriately adds no parameter details, avoiding redundancy. A baseline of 4 is applied since it doesn't need to compensate for any schema gaps.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('Returns') and resource ('full hash-chained audit trail of all compliance checks'), making it immediately understandable. However, it doesn't explicitly distinguish this tool from its siblings like 'verify_log', which might also deal with audit logs, leaving room for ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'check_action' or 'verify_log'. It lacks context about prerequisites, timing, or scenarios where this tool is preferred, leaving the agent to infer usage from the purpose alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• set_rulesB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions the syntax but fails to explain critical behaviors: whether this is a destructive overwrite or additive update, if it requires specific permissions, or what happens on success/failure. This leaves significant gaps for a tool that likely modifies system state.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is highly concise with two sentences that directly explain the tool's function and syntax. Every word contributes to understanding, with no redundant or vague phrasing, making it efficiently front-loaded and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations and output schema, the description is incomplete for a tool that sets rules. It omits essential context: expected outcomes, error conditions, side effects, and how it interacts with sibling tools. For a mutation tool with no structured safety hints, this leaves the agent under-informed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents the 'rules' parameter as an array of strings. The description adds minimal value by providing example syntax ('forbid delete_user'), but does not elaborate on semantic rules, validation, or error handling beyond what the schema implies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Set covenant rules') and specifies the syntax format ('permit/forbid/require syntax'), making the purpose specific and actionable. It distinguishes from sibling tools like 'check_action' or 'get_audit_log' by focusing on rule configuration rather than verification or logging.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like 'check_action' for verifying rules or 'verify_log' for audit purposes. The description lacks context about prerequisites, such as whether rules are applied immediately or require validation, leaving usage unclear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• verify_logB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool 'verifies integrity' and 'detects tampering,' which implies a read-only, diagnostic operation, but doesn't clarify aspects like performance impact, error handling, or what happens if tampering is found. For a tool with zero annotation coverage, this leaves significant gaps in understanding its behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences with zero waste: 'Independently verify the integrity of the hash-chained audit log. Detects any tampering.' It is front-loaded with the core purpose and efficiently adds the outcome. Every sentence earns its place by providing essential information without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (verification of a hash-chained log), lack of annotations, and no output schema, the description is minimally adequate. It states what the tool does but doesn't cover behavioral details like output format, error cases, or side effects. For a diagnostic tool with no structured support, it meets the bare minimum but leaves gaps in completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters, and schema description coverage is 100%, so there are no parameters to document. The description doesn't need to add parameter semantics, but it appropriately avoids mentioning any. A baseline of 4 is applied since no parameters exist, and the description doesn't mislead about inputs.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Independently verify the integrity of the hash-chained audit log. Detects any tampering.' It specifies the verb ('verify'), resource ('hash-chained audit log'), and outcome ('detects any tampering'). However, it doesn't explicitly differentiate from sibling tools like 'get_audit_log' or 'check_action', which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention sibling tools like 'check_action' or 'get_audit_log', nor does it specify prerequisites, timing, or exclusions. The agent must infer usage from the purpose alone, which is insufficient for clear decision-making.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.