Server Quality Checklist

Tool Scores

• rollA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description takes on the full burden of explaining behavior. It clearly states that the tool returns 'the total result and a formatted output string' and notes this is for 'quick single rolls without needing detailed breakdown,' implying it does not expose individual dice. The extensive supported notation list also sets expectations for accepted inputs. However, it does not disclose failure behavior for invalid expressions, which would round out the transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long but well-structured with clear sections (Purpose, When to Use, Supported Notation, Examples). Every section earns its place given the tool's syntactic complexity. It is front-loaded with the core purpose, and the examples are illustrative rather than filler. A slight redundancy exists between 'Common Examples' and the notation list, but overall it remains efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  This tool has no output schema and no annotations, yet the description covers purpose, usage boundaries, supported syntax, typical examples, and distinctions from all three sibling tools. It tells an agent exactly what to expect (total result and formatted string) and how to construct valid expressions. For a tool with this many notation variants, the description is remarkably complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the parameters are already documented. The description adds significant value by providing a comprehensive list of notation examples and explaining how allow_comments works with a concrete example ('1d20 fire damage'). This enriches the bare schema definitions with practical usage context, going beyond the baseline of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Roll dice using standard RPG notation and return the total result.' It clearly distinguishes itself from siblings by naming roll_detailed, roll_batch, and validate_syntax as alternatives, and by emphasizing this is the 'most common tool for standard RPG scenarios.' This leaves no ambiguity about what the tool does.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description devotes dedicated WHEN TO USE and WHEN NOT TO USE sections, explicitly listing scenarios like attack rolls, ability checks, and damage calculations. It also names exact alternative tools for each exclusion, such as 'use roll_detailed instead' for individual die results. This exceeds basic guidance and gives an agent clear decision criteria.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• roll_batchA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It discloses that each roll is independent, failure behavior (returns error entry, others continue), success/error fields, and that results come in a structured array. This goes well beyond what a typical tool description provides.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long but extremely well structured with clear sections (PURPOSE, WHEN TO USE, WHEN NOT TO USE, ERROR HANDLING, COMMON EXAMPLES). It is front-loaded with purpose. Some examples are repetitive and could be trimmed, but every section contributes to decision-making and usage. Slightly verbose yet consistently effective.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  There is no output schema, so the description explains return values ('structured array', 'success', 'total', 'result', 'error'). It covers error handling, use cases, alternatives, and provides examples. Combined with the detailed input schema, this is a fully self-sufficient description.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%: expressions and allow_comments are already described. The description adds many examples for the expressions parameter, but does not add meaningful semantics for allow_comments beyond the schema. It provides useful context but does not heavily compensate since schema already covers both parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a clear, specific phrase: 'Roll multiple different dice expressions in a single efficient operation.' It distinguishes itself from sibling tools by explicitly contrasting with roll and roll_detailed, and the WHEN TO USE / WHEN NOT TO USE sections reinforce the unique batching behavior.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Dedicated 'WHEN TO USE' and 'WHEN NOT TO USE' sections provide explicit context and alternatives. It names roll for single rolls, roll_detailed for AST needs, and warns against dependent rolls. This is exactly the kind of guidance needed for tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• roll_detailedA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully carries the behavioral transparency burden. It explains the AST structure, individual die results with 'kept' flags, and provides examples for complex mechanics like rerolls and exploding dice, leaving no ambiguity about what the tool returns.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Though lengthy, the description is well-organized with clear section headers and notable front-loading. Every section contributes to understanding, and the examples earn their place. It is appropriately detailed for a tool with complex output.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity and lack of output schema, the description is exceptionally complete. It covers the AST tree fields, individual die result format, and common examples, fully preparing an agent to use the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds value beyond the schema by explaining the expected expression format and providing concrete examples (e.g., '4d6kh3', '2d20kh1') that clarify how parameters should be used in context.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the tool as rolling dice with a comprehensive breakdown including AST structure and individual die values. It distinguishes itself from siblings by emphasizing detailed output, making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The 'WHEN TO USE' and 'WHEN NOT TO USE' sections explicitly state appropriate contexts and name alternative tools (roll, roll_batch) for different needs. This provides crystal-clear guidance on tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• validate_syntaxA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Even without annotations, the description fully discloses behavior: it is read-only, produces no random results, and explicitly lists what it checks and does NOT check (e.g., logical sense like '1d0' passing, performance implications). This gives the agent a complete understanding of the tool's capabilities and limitations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the purpose in the first sentence and uses clear section headings for organization. It is somewhat long for a tool with one parameter, but every section adds value through examples and use cases, so it earns its length. Slight deduction for verbosity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is complete for a simple validation tool: it covers purpose, usage guidelines, validation scope, examples, and real-world use cases. Since there is no output schema, the description doesn't need to explain return values, but it fully compensates for the lack of annotations and provides all necessary context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema only says 'Dice expression to validate', but the description provides extensive parameter semantics through valid/invalid examples (e.g., '1d20+5' valid, '1d' invalid) and explains what constitutes valid syntax. This goes far beyond the schema's minimal description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description immediately states the tool validates dice expression syntax without rolling dice, which is specific and unambiguous. It clearly distinguishes this from sibling roll tools by emphasizing the non-rolling, read-only validation purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit 'WHEN TO USE' and 'WHEN NOT TO USE' sections, naming alternatives like roll, roll_detailed, and roll_batch. It lists concrete use cases such as user input validation and form validation, making it easy to decide when to invoke this tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.