Server Quality Checklist

Tool Scores

• list_documentsA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It explicitly discloses that this tool queries the cache only, makes no API calls, requires a populated cache, returns a list with specific fields, and sorts by entity count. It does not mention behavior on empty cache or invalid source values, which would add further transparency, but it covers the main behavioral traits well.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a one-sentence purpose, usage explanation, examples, Args section, Returns section, and a Note. It is front-loaded with the primary purpose and uses concise headings; no sentence is wasted.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list tool with one optional parameter and no annotations or output schema provided, the description thoroughly covers purpose, prerequisites, parameter values, return format, and sorting behavior. It leaves no major gaps for an agent to invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema only defines 'source' as a nullable string with no description. The description compensates by listing valid values ('open5e_v2', 'orcbrew', None), explaining what each source refers to, and providing examples. However, the intro mentions a third data source (D&D 5e API) that is not reflected in the source values, leaving slight ambiguity about how to filter those documents.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb and resource: 'List all available D&D content documents in the cache.' It clearly distinguishes this tool from sibling search tools by explaining it is for discovering available source documents, not searching content. It also states the scope (cache) and purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context: 'Use this to see which books, supplements, and homebrew content you have access to, then use the documents parameter in other tools to filter content.' It also explains the prerequisite (populate cache) and that it makes no API calls. However, it does not explicitly name alternate tools or exclusion cases, so it lacks the explicit when-not guidance seen in top-tier examples.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_ruleA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses semantic vector search behavior, hybrid search combining filter and semantic queries, a default limit of 20 for performance, and raises ValueError/APIError. It also mentions internal repository/caching patterns, which adds some non-behavioral context but still demonstrates transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long but well-structured with clear sections for purpose, examples, args, returns, and raises. It front-loads the core purpose and uses bullet-style examples. The repeated semantic search examples are somewhat redundant, but the length is largely justified by the 5-parameter complexity and 10 rule_type variants.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Even with an output schema present, the description provides per-rule_type return structures, error conditions, filtering capabilities, and multiple search modes. It covers all practical invocation scenarios an agent might encounter, making the tool independently understandable without further documentation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate for all five parameters. It does so meticulously: rule_type is fully enumerated with plain-English meanings, section is scoped to 'rule' only, documents are tied to list_documents(), search is explained as semantic, and limit includes a default and performance rationale. The extensive examples reinforce parameter usage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Look up D&D 5e game rules, conditions, and reference information,' naming the specific verb, resource, and scope. The rule_type enum and detailed categories clearly differentiate it from sibling tools like search_spell or search_creature.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Extensive examples show when to use each rule_type, filtering by section/documents, and semantic/hybrid search modes. It states this is 'Essential for resolving rules questions during play or character building.' However, it never explicitly mentions when to prefer this over search_all or other sibling search tools, though the categories imply it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_character_optionA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It thoroughly discloses caching behavior ('Results are cached... repository pattern'), explains the first-call vs. subsequent-call semantics, and details the _repository_context injection for testing. It also lists error conditions (ValueError, ApiError), making the tool's behavior highly transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is quite long, but it is well-structured with clear sections: opening summary, caching note, examples, Args, Returns, and Raises. Each section adds value, though some parts (e.g., exhaustive list of class names) could be shortened without losing essential information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is complex with 4 parameters and variable return structures by type. The description provides comprehensive coverage: per-type return structures, error handling, examples for all parameter combinations, and a pointer to list_documents(). Despite having an output schema, the description still adds crucial context and makes the tool fully self-contained.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema coverage, the Args section is essential and excellently detailed: type explains each enum value with examples and class lists, documents references list_documents() and gives examples, search explains semantic/vector search, and limit explains its default and use. This fully compensates for the lack of schema-level descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Retrieve D&D 5e character creation and advancement options' and explicitly lists classes, races, backgrounds, and feats. This clearly distinguishes it from sibling search tools like search_spell or search_creature by focusing specifically on character options.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains typical use cases ('character creation and level-up decisions') and provides detailed examples for default, semantic, and hybrid search. It references list_documents() for filtering, but does not explicitly state when not to use this tool or mention alternative tools like search_spell, so while clear, it lacks explicit exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_equipmentA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It transparently specifies that search uses 'vector similarity' for semantic queries, automatically caches via the repository, can 'return many results' for type='all', and that limit applies to total results. It also details return structures per equipment type and lists the ApiError exception. This exceeds typical disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long but well-organized with Examples, Args, Returns, and Raises sections. It front-loads the purpose and uses code examples effectively. While some examples are repetitive, the overall structure makes it easy to scan, and the length is justified by the tool's 14-parameter complexity. A slightly more condensed version would be ideal, but it remains appropriately sized.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (14 parameters, no annotations, no output schema in structured form), the description is remarkably complete. It covers purpose, usage patterns, parameter semantics, return value structures for each type, default behavior, and error handling. The Returns section compensates for the lack of an output schema, and the Raises section clarifies failure modes.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate, and it does thoroughly. Every parameter is explained with concrete meanings, examples, and caveats. For instance, is_simple lists which weapons are simple, cost_min/max are defined as gold pieces, and search explains semantic vs. exact matching. This adds substantial value beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Search and retrieve D&D 5e weapons, armor, and magic items using the repository pattern,' providing a specific verb, resource, and scope. It clearly distinguishes itself from sibling tools like search_spell and search_creature by focusing on equipment categories. The detailed filters and example queries reinforce the tool's purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides extensive usage context through examples showing various filter combinations, semantic search, and hybrid search. It explains defaults like type='all' and limit=20, and indicates when to use different parameters. However, it does not explicitly contrast with sibling tools or state when NOT to use this tool, making it clear but not fully prescriptive.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_spellA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full responsibility for behavioral disclosure. It goes beyond basics by explaining the caching mechanism ('First call: Fetches from API and caches in database; Subsequent calls: Returns cached results'), test context injection via _repository_context, and the potential ApiError on network failures. It also documents semantic search behavior and return fields.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (summary, caching, examples, args, returns, raises) and is front-loaded with a summary. However, it is quite verbose, with multiple repetitive examples for similar filter types (e.g., five damage_type examples). While the detail is helpful for a complex tool, it could be trimmed slightly without losing value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (12 parameters, no annotations, and an output schema that is not detailed in the context), the description is fully complete. It covers all parameters with semantics and examples, explains the return structure field-by-field, documents error behavior, and discloses caching and test injection. Nothing critical is missing for an agent to correctly invoke this tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must fully compensate. It does so with detailed explanations for all 12 parameters, including valid values (e.g., level 0-9, school list), inclusive bounds for level_min/level_max, examples for casting_time, and clarification that 'damage_type' is new in Phase 3. The documents parameter even points to list_documents(). This far exceeds what the bare schema provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Search and retrieve D&D 5e spells using the repository pattern.' It uses a specific verb+resource combination and mentions comprehensive filtering, distinguishing it from sibling tools that search other D&D data types (creatures, equipment, rules, etc.). The scope is unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides extensive usage guidance through examples and parameter explanations, covering searching by level, school, class, damage type, and combining filters. It also references list_documents() for the documents parameter. However, it does not explicitly state when NOT to use this tool or contrast it with alternatives like search_all, so it falls short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_allA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully discloses behavioral traits: semantic matching is always enabled, typo tolerance, post-filtering by document, return structure ('type' or 'model' field), and raising of ApiError. This is comprehensive and provides context beyond what annotations would offer.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a description, examples, args, returns, and raises sections. It is longer than average but every sentence and example contributes value—there is no filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (4 params, 1 required, varied output types, no annotations), the description covers all essential aspects: use cases, parameter semantics, return structure, and error handling. It even includes examples for different filtering scenarios, making it fully complete for an agent to use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, but the description's Args section explains every parameter in detail: query handles typos and concepts, content_types lists allowed values and default, documents explains post-filtering and gives examples, limit states default. This significantly adds meaning beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool 'Search across all D&D content with semantic matching,' using a specific verb and resource. It distinguishes itself from sibling tools (search_spell, search_creature, etc.) by covering multiple content types simultaneously, which is explicitly mentioned.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear when-to-use guidance: 'Perfect for exploratory searches' and 'when you're not sure of exact spelling.' While it doesn't explicitly say 'use search_spell for exact spell searches,' the contrast with sibling tool names and the emphasis on cross-entity search imply the alternative. No explicit exclusions are stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_creatureA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description bears full responsibility for behavioral disclosure, and it delivers: it mentions caching through the repository, semantic/vector search behavior, result content, an ApiError raise condition, and even a test context injection mechanism. This goes well beyond a basic search tool description and gives the agent a strong mental model of side effects and guarantees.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long but highly organized into clear sections: overview, examples, Args, Returns, and Raises. Every line adds practical value—examples illustrate parameter combinations, and the Args section is exhaustive. The structure makes the length easy to parse, and no redundant filler is present.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 10 parameters, zero schema-level descriptions, no annotations, and a high-complexity search API, this description is exceptionally complete. It covers all parameter semantics, return value structure, error behavior, caching, and provides both basic and advanced usage scenarios, leaving no significant gap for an agent to make a wrong invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description compensates with detailed Args for all 10 parameters, including valid values (e.g., creature types, size categories), example ranges, defaults, and guidance for combining filters. It even explains fractional CR values and how to use documents with list_documents(), adding meaning far beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Search and retrieve D&D 5e creatures using the repository pattern,' clearly identifying the verb, resource, and implementation context. It distinguishes itself from siblings like search_spell and search_equipment by focusing specifically on creature lookup with stat blocks and combat statistics.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides extensive, concrete usage examples for all parameter combinations, including semantic search, hybrid search, and document filtering. It does not explicitly name sibling tools as alternatives or state when not to use this tool, so it misses the 'when-not/alternatives' bar for a 5, but the context is very clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.