Server Quality Checklist

Tool Scores

• erdl_simulateA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Description implies it is a safe, non-destructive simulation ('test... BEFORE creating'), but without annotations it does not explicitly state it is read-only or has no side effects. Additional statements about safety would improve transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences with front-loaded purpose and clear flow. Efficient and easy to parse, though it could be slightly more concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers the essential workflow: test before creation, show results, ask to proceed. However, it does not explain what the '3 scenarios' are or how they are generated, leaving minor ambiguity for the agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, and the description does not add extra meaning beyond what is already in the parameter descriptions. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description specifies 'Test a potential rule against 3 scenarios BEFORE creating it,' providing a specific verb ('test'), resource ('rule'), and scope ('before creation'). This clearly differentiates it from the sibling tool erdl_create_rule.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit instructions: 'Always call this BEFORE erdl_create_rule when the user says "remember this" or "create a rule".' It also tells the agent to show results and ask for confirmation, leaving no ambiguity about when and how to use the tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• erdl_create_ruleA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must convey behavioral traits. It states the rule is saved to ~/.openoba/rules/ and takes effect immediately with no restart needed. While it could mention potential side effects like overwriting existing rules, the provided information is adequate for understanding the basic behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a clear purpose statement followed by relevant example scenarios. It is concise (three short paragraphs) and front-loaded, though the examples could be slightly trimmed. No unnecessary information is present.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 6 parameters (4 required), no output schema, and no annotations, the description covers the core use case and provides examples. However, it lacks information about return values (e.g., rule ID or success message) and error conditions. This is a minor gap, resulting in an adequate but not comprehensive description.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so baseline is 3. The description provides example scenarios that map natural language to category and decision, but does not add detailed semantics beyond what the schema provides. The examples help contextualize parameter usage, which justifies the baseline score.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Create a new ERDL rule from natural language.' and provides specific example scenarios that illustrate the tool's purpose. This distinguishes it from sibling tools like erdl_evaluate, erdl_explain, erdl_list_rules, and erdl_simulate.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Use this when the user corrects your behavior and wants you to "remember" it.' This provides clear guidance on when to invoke this tool versus alternatives, and the example scenarios further illustrate appropriate usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• erdl_evaluateA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description provides adequate behavioral insight: it evaluates rules, returns a decision, and does not execute the tool itself. It could add that no state is modified, but the core behavior is clear.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is longer than necessary, including a full response format with badges and detailed decision handling. While well-structured, it could be more concise by moving the response format to a separate section or simplifying it.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 4 parameters, one required, and nested objects, the description covers the essential usage. It lacks details on error cases (e.g., no rules loaded) but is sufficient for basic operation. The output schema is absent, so the description compensates with response format details.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so baseline is 3. The description repeats the parameter names but does not add meaningful meaning beyond the schema's brief descriptions. For example, 'tool_args' is described as 'arguments being passed to the tool', which is obvious.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states that the tool evaluates a planned tool call against loaded rules before execution. It distinguishes from sibling tools like erdl_create_rule, erdl_explain, etc., which handle rule management rather than evaluation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'YOU MUST call this BEFORE every tool call' and provides clear instructions for handling each decision outcome (ALLOW, DENY, CORRECT, REQUEST_HUMAN). This gives unambiguous guidance on when and how to use the tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• erdl_explainA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, but description discloses the tool shows 'every rule that was checked and whether it fired' and references 'the last action', giving insight into internal state retrieval. Missing limitations (e.g., requires prior evaluation) but sufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Short, front-loaded with purpose, bullet points for usage. Every sentence serves a purpose; no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  No output schema, but description does not explain the return format or structure of the decision trail. Mentions 'shows every rule' but not how it's presented. Incomplete for a complex tool with nested objects.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema covers 100% of parameters. Description adds value by linking tool_name and tool_args to 'same as you used for erdl_evaluate', providing cross-tool context. No additional format details, but helpful.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states 'Show the FULL decision trail for the last action' and answers 'why did you do that?'. It distinguishes from siblings like erdl_evaluate (which runs evaluation) by focusing on explanation after an action.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly lists when to use: when user asks why, confused about DENY/unexpected ALLOW, or for transparency. No explicit exclusion, but context is clear and actionable.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• erdl_list_rulesA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description bears the full burden. It describes a read-only listing operation with no mention of side effects, which is appropriate for a list action. The behavioral impact is minimal and predictable, fulfilling transparency needs adequately.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise (three sentences) with no wasted words. Each sentence adds value: purpose, usage guidance, and categorization hint. Structure is logical and front-loaded with the core action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one optional parameter, no output schema), the description covers all essential aspects: what it does, when to use, and how to categorize results. There is no significant missing information.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description adds a categorization hint ('Categorize by type: ...') that mirrors the enum in the schema but does not provide additional meaning beyond what the schema already offers.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action (list) and resource (ERDL rules). It includes usage examples ('what rules do you have?') and naturally distinguishes from sibling tools like erdl_create_rule or erdl_evaluate which have different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit guidance on when to use the tool (when user asks about rules or constraints). Lacks explicit when-not-to-use or alternative tools, but the context of sibling names and the examples make the usage scope clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.