Server Quality Checklist

Tool Scores

• check_kbA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare the tool as read-only, idempotent, and non-destructive. The description adds behavioral details: it is cheap, backend-free, and catches specific errors and warnings, enriching the agent's understanding beyond annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and concise, starting with the core purpose, followed by usage guidance, error types, and return fields. Every sentence adds value without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple parameter schema (one string) and detailed description of the output (KBCheckResult fields), the description provides complete information for an agent to use the tool effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description does not elaborate on the single 'knowledge' parameter, but the input schema provides a thorough explanation of the format. Since the schema covers the parameter adequately, a baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool validates a knowledge base statically without running it, and distinguishes it from 'reason' by recommending calling it before reasoning. This is specific and differentiates from siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly advises using this tool before 'reason' after editing a knowledge base, providing clear context. It does not include exclusions, but the usage context is well-defined.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• diagnoseA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and destructiveHint=false, and the description adds behavioral context by explaining the three modes, return format (DiagnosisResult), error handling (ok and error fields), and limitations like bounded assumptions. No contradictions with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a clear opening statement, followed by mode explanations and return field details. Every sentence serves a purpose, and the information is front-loaded. No redundant or fluff content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (diagnosis with multiple modes, specific input language, and structured output), the description covers all necessary aspects: when to use, mode details, input format, and output fields (including error conditions). Output schema exists but the description explains the fields for clarity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema includes descriptions for all parameters, so baseline is 3. The description adds value by elaborating on query syntax, knowledge base format, and mode behavior beyond the schema's brief descriptions. For example, it explains variables with $, conjunction with AND, and the max assumptions for what_needs mode.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Explain why a query holds, why it fails, or what would make it true.' It also distinguishes from siblings by explicitly referencing the 'reason' tool and advising to use when 'reason returns something you did not expect.'

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Reach for this when reason returns something you did not expect,' providing clear context for when to use. It also details three modes with specific use cases. However, it does not explicitly mention when not to use it or compare with other siblings like what_if or check_kb.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• reasonA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses detailed behavioral traits beyond the annotations (readOnlyHint, idempotentHint, etc.): it explains the return structure (ReasonResult with ok, solutions, proof, etc.), closed-world semantics (empty solutions with ok:true means false), query source (from parameter or knowledge lines), and truncation behavior. No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured: a one-sentence core purpose, followed by usage scenarios, parameter interaction note, and a thorough yet concise breakdown of the return type. Every sentence is substantive, with no redundancy. The most critical information is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (deduction with proofs) and the presence of a detailed output schema, the description provides sufficient context: it explains the return fields, error handling, truncated results, and the meaning of empty solutions. It does not repeat schema content but adds essential usage context. Output schema exists, so return value details are adequately covered.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already has detailed descriptions for each parameter. The description adds value by explaining the relationship between query and knowledge parameters ('The goal comes from the query parameter, or from the ? ... line(s) in knowledge when query is omitted'), which is not present in the schema descriptions. This enriches parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Prove a goal against a knowledge base and return every solution with its proof,' providing a specific verb+resource. It further lists example use cases (transitive relations, permission inheritance, etc.) and implicitly distinguishes from siblings like 'diagnose' and 'what_if' by emphasizing multi-step deduction.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states 'Use this for any question that needs multi-step deduction over facts and rules,' giving clear positive guidance. However, it does not explicitly state when not to use it (e.g., 'do not use for simple fact lookups'), which would strengthen differentiation from siblings like 'check_kb'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• what_ifA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description reveals detailed behavior beyond annotations: it returns a `WhatIfResult` with fields like `ok`, `before_count`, `after_count`, `delta`, `solutions_before`, `solutions_after`, `applied`, `conclusion`, and `error`. It confirms the tool is read-only (readOnlyHint=true), idempotent, and non-destructive, and adds that only facts can be mutated.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is moderately long but well-organized. It begins with the core purpose, then usage instructions, modification format, and finally a full list of return fields. Each section adds value without redundancy. Minor improvement: the return fields could be more compact, but overall it is efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is complete for a complex tool with 5 parameters (all with 0% schema coverage) and an output schema. It covers input syntax, error conditions, and all return fields. There are no gaps; the agent can use this tool correctly based solely on the description.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Although the input schema has 0% description coverage, the description compensates fully. It explains the `modifications` syntax (`+ fact(...)` / `- fact(...)`), the `query` format (Euclid-IR body syntax), and the `base_knowledge` format (text or YAML with facts, rules, queries). It also notes that variables start with `$`, `_` is a wildcard, and `NOT` is closed-world negation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a clear action: 'Apply hypothetical fact changes and compare the query before and after.' It explains the tool's value (test without committing) and gives concrete examples (promoting a user, adding a resource, revoking an assignment). The purpose is distinct from sibling tools like 'reason' and 'diagnose' which focus on non-hypothetical reasoning.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use the tool ('test a change without committing to it') and provides examples. It also clarifies limitations ('Only facts can be changed, not rules') and warns about errors ('A `-` line that matches no existing fact is an error'). Although alternatives are not named, the usage context is clear and well-documented.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.