Server Quality Checklist

Tool Scores

• conflictsB
  3.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full transparency burden. It conveys that the tool is read-only in nature ('Open' and 'Empty is the normal state') and manages expectations about empty results. However, it does not disclose any side effects, required permissions, response structure, or error conditions, leaving significant gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise—two short sentences that deliver core purpose and an important behavioral caveat ('Empty is the normal state'). Every word adds value, and the structure is front-loaded with the primary action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has no output schema and sparse annotations, so the description must explain return values and expected behavior. It only says 'Open contradictions' without specifying the structure of each conflict, how 'limit' affects output, or whether pagination exists. This leaves an agent under-informed for a tool with one parameter and no output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description does not mention the 'limit' parameter at all. The schema only provides a type and default, so the agent gets no meaningful semantics beyond the parameter name. Since the tool has one parameter, this is a notable omission, though 'limit' is somewhat self-explanatory.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: 'Open contradictions the compiler could not settle.' This clearly distinguishes it from siblings like 'recall' or 'timeline' by focusing on compiler conflicts. However, 'open' is slightly ambiguous (display vs. fetch), so it doesn't quite reach a 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives implied usage context by stating 'Empty is the normal state,' suggesting it is used for checking unresolved conflicts. However, it provides no explicit when-to-use or when-not-to-use guidance, nor does it mention any alternatives or sibling tools. Thus, it is clear but lacks exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• statsB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are present, so the description carries the full burden. It implies a read-only operation (returning counts) and is not misleading, but it does not explicitly state output format, access requirements, or limits. This is a moderate level of transparency for a simple stats tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that immediately conveys the core value. It is concise with no redundant information, earning a perfect score.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  As a zero-parameter tool with no output schema, the description covers the general purpose but lacks explicit return structure and usage guidance. It is adequate for the simplicity but not fully comprehensive, leaving some contextual gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the empty schema is complete and the baseline of 4 applies. The description adds useful context about the dimensions of the counts (source, kind, trust), which enriches understanding of the result.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly indicates the tool provides counts of the corpus by source, kind, and trust, giving a specific resource and scope. It distinguishes from siblings (recall, timeline, conflicts) by focusing on aggregate statistics, but lacks an explicit action verb.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus sibling tools, nor any mention of context, prerequisites, or exclusions. The description only states what it returns, leaving usage entirely implied.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• timelineA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It reveals useful behavior: matching claims, assertion order, and showing which claims superseded others. However, it does not mention side-effect status, authentication requirements, or return format, so some behavioral ambiguity remains.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise: two short sentences. It front-loads the core purpose, then adds behavior and usage guidance without filler or repetition. Every clause earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple two-parameter, read-style tool with no output schema, the description provides solid coverage: purpose, mechanism, and when-to-use. It lacks explicit return value details and limit behavior, but given the low complexity, it is reasonably complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate for parameter meaning. It conceptually references 'subject' but gives no format or expected values, and 'limit' is completely ignored. The description adds minimal value beyond the schema's bare parameter names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states what the tool does: show how belief about a subject changed over time. It adds specificity with 'matching claims in assertion order, showing which superseded which,' which distinguishes it from sibling tools like 'conflicts' by focusing on temporal evolution. However, it lacks an explicit verb like 'returns' or 'lists' and does not directly name sibling alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly gives usage context: 'Use when an answer may have changed rather than merely being unknown.' This tells the agent when the tool is appropriate. It does not mention exclusions or explicitly compare to sibling tools, but the guidance is clear enough for a 4.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• recallA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully carries the transparency burden. It discloses how claims are ranked, what fields are returned, how as_of changes behavior, and how intent can override the default reading—offering a clear behavioral model.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded and tightly structured. The short opening sentence states purpose, the warning section earns its place by preventing a real misuse, and the parameter notes are compact and useful.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description covers purpose, query formulation, ranking behavior, return fields, and key parameters. It is nearly complete, but the lack of explanation for k and include_superseded, plus no detailed return format, prevent a perfect score.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so description compensation is critical. It thoroughly explains query, as_of, and intent, adding meaning beyond the schema. However, k and include_superseded are left unexplained, a modest gap considering the useful context provided for the other parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Answer a question from the claim graph.' It clearly differentiates recall from sibling tools by emphasizing ranked question answering over timeline, conflict, or stats navigation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives strong usage guidance with 'PASS THE QUESTION, NOT KEYWORDS,' concrete examples, and when to use as_of or intent. However, it does not explicitly mention sibling tools or state when not to use recall, so it misses explicit exclusion/alternative guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.