Server Quality Checklist

Tool Scores

• diagnoseC
  2.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description must disclose behavioral aspects itself. It only mentions the diagnostic purpose and modes, but omits any side effects, resource implications, authentication needs, or what the output contains. This is a significant gap for a diagnostic tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely short (one sentence), but it omits critical parameter explanations. While concise, it under-specifies the tool's usage, making it ineffective. The structure lacks a clear breakdown of modes and parameters, so it does not serve as a useful guide.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 7 parameters, no annotations, and no schema descriptions, the description is grossly incomplete. It fails to explain what each parameter does, what the output contains, or how the tool behaves in different contexts. The presence of an output schema does not compensate for missing parameter semantics.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage Dominic, and the description does not explain any of the 7 parameters except 'mode' implicitly. Parameters like kb_id, query, knowledge, max_depth, max_solutions, and delta_knowledge are left undefined, making it impossible to use the tool correctly without external knowledge.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's action ('Diagnose') and resource ('why a query succeeds or fails'), and lists specific modes that further clarify purpose. However, it does not explicitly differentiate from sibling tools like 'explain' or 'what_if', so it loses some points on distinction.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description lists modes but provides no explicit guidance on when to use this tool versus alternatives like 'explain' or 'what_if'. It does not state prerequisites, typical scenarios, or exclusions, leaving the agent without clear selection criteria.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• reasonC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of disclosing behavioral traits. It only mentions that it returns proof trees but does not indicate side effects, performance costs, or whether it modifies the knowledge base. The description is too sparse to inform the agent about potential impacts.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no redundancy. It efficiently conveys the main action and result format. However, its brevity veers toward under-specification for a complex tool, though this is more a completeness concern than conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has six parameters, no annotations, and an output schema, the description is far from complete. It does not mention what inputs are required, how the reasoning process works, or any constraints. Even though an output schema exists, the description leaves the tool's usage and behavior largely unexplained.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has zero description coverage, and the description does not explain any of the six parameters (kb_id, query, knowledge, max_depth, max_solutions, delta_knowledge). Without any parameter explanation, the agent cannot understand how to construct a valid call, so the description fails to compensate for the lack of schema details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: performing logical deduction on a knowledge base and returning solutions with proof trees. This is specific and distinct from sibling tools like explain, diagnose, and what_if, which likely cover other analytical tasks.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives, nor any prerequisites or exclusions. It does not mention how this differs from diagnose or what_if in terms of use cases, leaving the agent without context for selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_kbC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It mentions the checks performed but does not disclose whether the tool modifies anything (likely read-only), what happens on failure, or whether it requires an existing KB vs. inline knowledge. It does not contradict annotations (none exist), but it is thin on behavioral details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, concise and front-loaded with the main purpose. It lists specific checks efficiently. However, it could be slightly more structured (e.g., separating purpose from usage), but it is not verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 3 parameters with 0% schema coverage, no annotations, and an output schema (not described), the description is incomplete. It does not explain the parameters, the output format, or the context of use (e.g., when to use delta_knowledge). The description is adequate for a simple check but lacks essential details for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description does not explain the three parameters (kb_id, knowledge, delta_knowledge). It does not clarify how they relate (e.g., whether kb_id is required, or how knowledge and delta_knowledge are used). The description adds no meaning beyond the schema's property names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: checking a knowledge base for consistency, listing specific checks (syntax errors, undefined predicates, circular rules, duplicates). It distinguishes from siblings like diagnose or explain by focusing on consistency checks, though it doesn't explicitly name alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for validating a knowledge base but does not specify when to use this over siblings like diagnose or what_if. It lacks explicit guidance on when to use this tool versus alternatives, and does not mention prerequisites or context (e.g., whether kb_id or knowledge must be provided).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• what_ifC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden for behavioral disclosure. It explains the +/− syntax but does not say whether changes are permanent, what happens to the KB, or what output to expect beyond 'see how they affect query results.'

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded, with the core purpose in the first phrase and useful syntax guidance in the second sentence. It contains no filler, though it could have been longer to cover more behavioral detail without harming structure.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 7 parameters, no annotations, and minimal schema descriptions, the description is too sparse for reliable invocation. It gives one hint about modifications but leaves the roles of other parameters and the non-obvious semantics of the tool largely unexplained.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 7 parameters with 0% description coverage, and the description only clarifies the modifications parameter via the +/− syntax. Parameters like base_knowledge, delta_knowledge, max_depth, and max_solutions receive no semantic explanation beyond their names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: applying modifications to a knowledge base to see effects on query results. It is distinct enough from siblings like reason and explain, though it does not explicitly name or contrast them.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'What-if analysis' implies hypothetical exploration, and the +/− prefix instructions signal how to use it. However, there is no explicit guidance on when to use this tool versus siblings, nor when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• unregister_kbA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It does disclose the behavior when the kb_id is not registered (returns 'removed': false), which is useful. However, it omits details like whether removal is permanent, requires permissions, or has irreversible effects beyond the name.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, immediately states the purpose, and includes the key return behavior. There is no wasted wording or redundant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter removal tool, the description adequately covers the core action and a key edge case (missing kb). It could mention side effects or prerequisites, but given no output schema and simple behavior, it is reasonably complete for the sibling context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It adds minimal semantics by calling it a 'named knowledge base', implying kb_id is a name, but it doesn't explain the format, constraints, or how to find valid values. The description offers little beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Remove') and the resource ('named knowledge base from the registry'). It distinguishes from siblings like register_kb, list_kbs, and check_kb by specifying the unregister action.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description offers no guidance on when to use this tool versus alternatives, such as when to call unregister_kb instead of check_kb or register_kb. No context or exclusions are given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• explainA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations available, the description carries the full burden. It accurately discloses the core behavior—walking proof trees, producing natural-language steps, and citing rule IDs when present. It does not contradict any annotations (there are none). Missing details include side effects (likely none, but unstated) and behavior when there are no solutions, but the description gives a strong sense of what happens internally.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the verb and purpose, and every phrase adds value. There is no redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given there are 6 parameters, no annotations, and an output schema exists, the description is adequate for understanding the tool's primary purpose and output, but incomplete regarding parameter roles and usage context (e.g., how kb_id relates to knowledge, when to vary max_depth vs. max_solutions). The output schema covers return values, so that is not a gap, but the parameter and selection guidance are thin.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate for parameter meaning. It adds context for 'query', implicitly for 'max_depth' and 'max_solutions' via 'walk the proof tree of each solution', but leaves 'kb_id', 'knowledge', and especially 'delta_knowledge' undefined. The description does not explain the relationship between these parameters or how they affect the explanation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Explain'), names the resource ('how a query is proven'), and specifies the method and output ('walk the proof tree of each solution and return readable reasoning steps'). It clearly distinguishes this from siblings like 'reason' (likely performs reasoning) and 'diagnose' (likely investigates problems) by focusing on natural-language explanation of proofs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly implies the use case: when a user needs a human-readable explanation or proof trace for a query. However, it does not explicitly name alternatives or state when not to use this tool (e.g., when one actually wants to run a query vs. understand its derivation).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• register_kbA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the behavioral disclosure burden. It honestly discloses that registration 'Overwrites an existing kb_id' and that the KB is validated with check_kb first. However, it does not mention failure modes, return values, or permissions, leaving some behavioral gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is exactly two sentences, both dense with information: the purpose/reference benefit and the overwrite/validation behavior. No filler or redundancy, and the most important verb and resource appear first.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given a simple two-parameter schema and no output schema, the description covers the essential aspects: naming/registering, overwriting, and validation. It lacks details on error handling if check_kb fails, but for a low-complexity registration tool, it is sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It clarifies that kb_id is a reference name and knowledge is the KB text, mapping to the two schema properties. It does not provide format, length, or additional constraints, but it does add meaning beyond the bare titles.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Register a named knowledge base under a kb_id' with the specific benefit of allowing later references instead of resending KB text. It also distinguishes itself from siblings by noting overwrite behavior and the relationship to check_kb, differentiating it from unregister_kb, list_kbs, and check_kb.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives an explicit usage context: 'so later calls can reference it instead of resending the KB text' and mentions the prerequisite validation via check_kb first. It does not explicitly name alternative tools or state when not to use, but the purpose and precondition provide clear guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_kbsA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must fully disclose behavior. It adds 'metadata only,' which clarifies the return type (no content), but it does not mention side effects, permissions, or other behavioral aspects. It's adequate but not comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that conveys all essential information with zero fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list operation with no parameters and no output schema, the description adequately covers what it does and what it returns (metadata fields). It doesn't need more, though it could mention that it returns all KBs or ordering, but that's minor.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are no parameters (0 params), so per rubric the baseline is 4. The description provides no parameter-specific details, but none are needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('List') and the resource ('registered named knowledge bases'), and specifies the scope (metadata only), which distinguishes it from siblings like register_kb and unregister_kb.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies a read-only listing operation, which contrasts with mutating siblings like register_kb and unregister_kb. However, it doesn't explicitly state when to use it vs alternatives or when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.