Server Quality Checklist

Tool Scores

• compare_capsulesC
  2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It does add one useful constraint ('does not infer causality'), which is genuinely informative about output interpretation. However, it doesn't explain what the structured difference contains, whether this is a read-only operation, what format the comparison takes, or any side effects—leaving the agent with limited behavioral understanding for a comparison tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely brief at one short sentence, which is genuinely concise. However, it is under-specified rather than efficiently complete—this teeters between concise and insufficient. The fact that it contains no redundant filler earns some credit, but the brevity comes at the cost of essential information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 2 string parameters at 0% schema coverage, no annotations, and no output schema, the description provides almost no context. The agent cannot determine what input format to supply, what the comparison output looks like, or how this differs from inspecting capsules individually. Among seven siblings with varied investigative functions, this level of description is severely inadequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate entirely. Neither parameter (capsule_a, capsule_b) is explained in the description beyond the generic inference from their names and types. There is no note on what format the capsule identifiers take (ID string, path, URL?), whether order matters, or whether they refer to specific version tags. Two fully undocumented parameters at 0% coverage require substantial compensation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose2/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description says 'Structured difference between two capsule versions' which clarifies the verb (compare) and resource (capsules), but it reads more like a restatement of the tool name 'compare_capsules' rather than adding new specificity. It's not entirely tautological, but it adds minimal differentiation from siblings like find_related_capsules or drill_down.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no guidance on when to use this tool vs alternatives. The single sentence provides no context about which scenarios call for comparing capsule versions, no mention of prerequisites, and no exclusions. The phrase 'does not infer causality' hints at one limitation but doesn't explain when it WOULD be the right choice among the prominent sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• report_savingsC
  2.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full behavioral burden. The description reads like a list of metrics presented, suggesting a read/report operation, but it doesn't disclose whether this is a mutation, whether it requires prior investigation state, how it computes the estimates, or what the output format looks like. For a tool with zero annotation coverage, this is insufficient behavioral disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a compact single sentence listing the report's contents. It's brief and gets to the point, but it's structurally a fragment describing metrics rather than a proper explanation. No waste, but also minimal substance—on the under-specification side rather than the concise-and-complete side.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 2 parameters, no output schema, no annotations, and 0% schema coverage, the description should do substantial explanatory work. Instead it lists four metric categories and stops. It doesn't explain the two parameters, what the report format is, when it's meaningful to call it, or what the relationship to sibling tools (find_related_capsules, compare_capsules) is. Given the context richness of sibling names, more is expected.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% and there are 2 parameters (team, since) with no documentation in the schema. The description doesn't mention either parameter at all, leaving the agent to guess what 'team' and 'since' mean—though they are inferable (team name, time window). With zero schema coverage and no param info in the description, the description fails to compensate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose2/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description describes what the report contains (avoided work estimates including capsule reuse, duplicate query avoidance, context compression, and token savings), but never states the tool's verb or action clearly. It's unclear whether this tool generates, fetches, or compiles these savings. The name 'report_savings' implies a reporting action, but the description only lists report contents without a clear verb, and doesn't distinguish it from sibling tools like drill_down or verify_claim.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is essentially no guidance on when to use this tool versus the siblings. It doesn't say things like 'use after investigation' or 'call to summarize savings.' The semantic content of the savings (capsule reuse, duplicate queries, context compression) gives some implicit context about usage, but there's no explicit when/when-not guidance or mention of alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• drill_downC
  2.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the behavioral burden. It discloses that results are redacted and field-limited, and that the response reports estimated source bytes examined (cost transparency). However, it doesn't mention auth needs, rate limits, or what happens on failure. The description adds meaningful value but leaves gaps for a cost-path tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, compact and front-loaded with the core purpose first. The second sentence about cost-path reporting is relevant. No wasted words, though the dense jargon ('bounded raw evidence', 'capsule') hurts accessibility slightly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 0% schema coverage, no annotations, no output schema, and 4 parameters (2 undocumented by any prose and only structurally named in schema), the description is too thin. It doesn't explain return format, how fields filtering works, or resolution mechanics. The cost-path framing is useful but insufficient for reliable tool selection and invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It gives zero parameter information beyond the schema: fields, consumer, capsule_id, evidence_id are all unexplained in prose. The description doesn't clarify what 'fields' restricts, what a 'consumer' represents, or relationship between capsule_id and evidence_id.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states it retrieves 'bounded raw evidence behind a reference (redacted, field-limited)', giving a specific verb+resource. However, the term 'reference' and 'capsule' are domain jargon not defined, and it doesn't clearly differentiate from get_capsule or find_related_capsules siblings. It does add the distinguishing cost-path detail.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'explicit cost path' implies it should be used when cost matters, but no explicit when-to-use or when-not-to-use guidance is given. It doesn't mention alternatives or exclusions. The guidance is implied rather than stated explicitly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_capsuleC
  2.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the behavioral disclosure burden. It adds two useful behavioral notes: the tool respects redaction policy, and it returns a bundle of elements (claims, evidence, freshness, consumers). However, it doesn't disclose what happens with the 'detail' parameter (summary vs full), how pagination/limits work, or whether it's a read-only operation. The redaction-policy mention is a notable positive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded with the action and key deliverable. The content list is slightly verbose (five claim types enumerated) but that enumeration is genuinely useful. No wasted words overall.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 3 parameters with 0% schema coverage and no annotations or output schema, yet the description fails to document the 'detail' and 'consumer' parameters. These likely control output verbosity and redaction context respectively—both critically relevant to an agent choosing invocation arguments. The redaction-policy mention signals this but doesn't explain how the agent should set parameters to control it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must explain parameters, but it only explains capsule_id implicitly via the subject of the fetch. The 'detail' parameter (default 'summary') and 'consumer' parameter are not described at all—no note on what detail levels exist or what 'consumer' means for redaction/context. This is a gap since the description itself references a redaction policy that is likely tied to the consumer parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves a capsule and enumerates its contents (situation, typed claims, evidence references, freshness, consumers). It uses a specific verb-plus-resource pattern ('Fetch a capsule'). However, it doesn't explicitly distinguish itself from sibling tools like find_related_capsules or compare_capsules, though the detailed content list does implicitly differentiate it.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no guidance on when to use this tool versus alternatives. It doesn't mention that this fetches a single capsule by ID (which distinguishes get vs find/compare), nor does it warn against using it when a related-capsule or comparison workflow is needed. The context ('Respects the capsule's redaction policy') is informative but not a usage guideline.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• refresh_capsuleC
  2.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses that this re-queries the backend and creates a new version, which implies mutation (creating a version). However, it doesn't disclose side effects beyond versioning, whether re-querying has cost/latency implications, or how the delta is formatted. The 'creates a new version' hint partially compensates for missing annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is short (two sentences) and front-loads the key behavior (return only what changed). The instruction-like phrasing is efficient. It's concise without being under-specified.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool that mutates state (creates a new version), has 4 undocumented parameters, and lacks an output schema, the description is underspecified. It doesn't explain what the delta format looks like, what triggers re-querying, or whether parameters are mutually required (e.g., capsule_id vs investigation_id). Compare_capsules sibling suggests comparison could be relevant, adding ambiguity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% and all 4 parameters have empty defaults. The description adds no meaning to focus, consumer, capsule_id, or investigation_id. With 0% coverage, the description should help clarify which parameters matter for the delta behavior, but it says nothing about them.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns a delta (only what changed since the latest version) and creates a new version by re-querying. This is a specific verb+resource (refresh capsule) with clear scope, though it doesn't explicitly distinguish from siblings like compare_capsules which could be confused for diff functionality.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool vs. get_capsule or compare_capsules. The delta behavior is described but there's no statement of when a user would need a refresh versus retrieving or comparing. With siblings like compare_capsules and get_capsule, explicit usage boundaries would help significantly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_related_capsulesA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description states 'Never merges automatically,' which is a useful behavioral guardrail clarifying that the tool only returns candidates without mutating state. However, with zero annotations and no output schema, the description remains thin—it doesn't disclose response format, pagination, ordering, rate limits, or what 'reason for the match' looks like. For an investigation tool with no annotation coverage, the behavioral disclosure is minimal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the verb+resource, zero wasted words. Efficient and scannable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 7 parameters (including a symptom_signature concept that is non-trivial), no output schema, and no annotations. The description mentions the match reason output and non-merging behavior, which helps, but leaves significant gaps: what constitutes 'related' for each matching dimension, how time window interacts with start/end, what a symptom signature means, and return format details. Given the schema/annotation poverty, more depth is warranted for a full 5, but the core purpose and key output are covered.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It names three matching dimensions (service/environment/symptom/time window) and mentions the output reason, which maps loosely to parameters. However, it does not explain the time-window semantics or interaction between start, end, and window_minutes, nor the symptom_signature format. It adds some value but under-compensates for the total 0% schema coverage across 7 params.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  'Find existing capsules matching a service/environment/symptom/time window' uses a specific verb and resource (capsules), lists the four matching dimensions, and the sibling context (start_investigation, get_capsule, compare_capsules) suggests this is a discovery/search tool distinct from fetching a single capsule. The purpose is clear and well-differentiated.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description states this finds candidates and never merges, which implies it's a pre-merge discovery step among the sibling tools. However, it does not explicitly say when to use this vs compare_capsules or get_capsule, nor provide exclusion criteria or clearly frame it as the discovery/pre-check tool. The guidance is implied rather than stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• start_investigationA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of disclosure. It does disclose important side effects: reuse without re-query, flagging, candidate return, and the 'never merged silently' guarantee. However, it does not disclose potential costs (e.g., whether this launches long-running queries, whether it has side effects beyond a read, or what the compaction/linking entails). For a tool that starts work on production systems, more disclosure would help.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single dense paragraph plus a code example. It is reasonably concise and front-loads the core purpose and key reuse behavior early. The inclusion of an inline example for `symptom_signature` is valuable. It could be trimmed slightly, but the content density is good and every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 10 parameters with 0% schema coverage and no output schema or annotations, the description does substantial work but leaves gaps. It explains the reuse/merge semantics and the signature format, but does not clarify what each parameter controls (especially start/end/window_minutes interplay, environment scope, symptom_kind options). For a tool managing production investigations, the agent needs more parameter grounding to invoke correctly, though the core question/services/signature trio is well handled.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It does explain `symptom_signature` with a concrete example format ('HTTP_502:checkout'), and implies the meaning of several parameters like `force_new`, `window_minutes`, `environment`, `consumer`, and `services` through context. However, several parameters (start, end, symptom_kind, consumer) are only hinted at or left implicit, meaning a single broad sentence carries much of the burden across 10 parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Begin or reuse') and the resource ('bounded production investigation'). It specifies the primary output (a 'compact, source-linked evidence capsule') and distinguishes itself from siblings by describing its unique behavior around reusing existing investigations and handling related-but-not-identical matches. This is specific and non-tautological.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit guidance on when reuse happens: identical investigations are reused and flagged, while related-but-not-identical candidates are returned for the agent to choose ('never merged silently'). It also explains the `force_new` semantic implication by contrasting against reuse behavior, indicating when a new investigation would be forced. This is strong behavioral guidance for a tool that could otherwise silently behave unexpectedly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• verify_claimA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It properly discloses this is a read/verification operation returning evidence, which is useful. However, it doesn't disclose behavioral traits like whether it requires pre-existing capsule state, mutability implications, or any rate-limit/auth constraints, leaving some gaps for a tool with no annotation support.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences with no wasted words. The first sentence delivers the core purpose and listed outputs; the second sentence states the use case trigger. Every element earns its place with clean line wrapping.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a read/verification tool with 3 simple params and no output schema, the description covers the essential purpose, scope (single claim), what evidence it returns, and the use case trigger. The description is largely adequate; minor gap is the undefined 'consumer' parameter and lack of clarification on relationship between the two required IDs.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate for the parameters. The three parameters (capsule_id, claim_id, consumer) are not individually explained semantics-wise in the description beyond the implicit 'one claim' context. The description adds the conceptual link between claim and capsule but doesn't clarify what 'consumer' means or how capsule_id vs claim_id relate specifically.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Show') with a clear resource ('supporting and contradicting source evidence') and explicitly scopes it to 'one claim'. It distinguishes itself by naming specific outputs (query fingerprint, time range, deep link) not mentioned by any sibling tools. The final sentence 'Use to challenge an answer' reinforces its distinct use case.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context on when to use this tool ('Use to challenge an answer') and conveys its verification/doubt-checking purpose, which distinguishes it from siblings like drill_down or get_capsule. However, it doesn't explicitly name alternative tools or state when NOT to use it, so it lacks explicit exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.