Server Quality Checklist

Tool Scores

• evaluate_displayA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already mark the tool read-only and idempotent; the description goes much further by explaining how multiple lawful displays are represented, that all options are equally correct, that 'shall' and 'may' lights are not merged, and that results are verbatim from a pre-release data package. No annotation contradiction.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long, but the tool is complex and nearly every sentence carries a distinct caveat or output detail. It is front-loaded with the purpose and returns, though the density of the prose makes it harder to scan than a bulleted structure would be.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given there is no output schema, the description thoroughly explains return fields, relation semantics, option numbering, optional additions, citations, coverage limits, and data stability. It lacks a concrete call example and does not discuss invalid/empty fact inputs, but is otherwise complete for typical use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description compensates by naming the core fact fields (propulsion, activity, position, length), referencing the facts data package, and adding the crucial semantic that the input describes a situation, not a fitted vessel, so equipment choices are not settled. Per-field value details are left to the input schema, which provides them.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states the tool evaluates every complete lawful set of navigation lights from a fact record under COLREGS, and enumerates the returned data (applied entries, exemptions, lawful_displays). This makes the tool's purpose unambiguous and distinguishes it from a generic rule-text or single-light tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives clear when-to-use context (COLREGS Part C lights at night, from a fact record) and explicit exclusions (no day shapes, sound signals, or Part B rules). It does not name sibling tools or explicitly route to alternatives, which prevents a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• lightA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the readOnly/idempotent annotations, the description discloses that the content is verbatim from a pre-release data package, that nothing is paraphrased, that the surface is a 0.0.x preview subject to change, and that it is not for navigation. This is substantial behavioral context not available from annotations alone.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is dense but front-loaded with the core purpose and output fields, followed by necessary caveats. A few caveats are repeated in spirit, but each sentence adds useful information for a tool with significant data-quality warnings.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having no output schema, the description enumerates the return fields, mentions the bearing convention for arcs, specifies coverage and exclusions, and warns about data maturity. For a single-parameter read-only tool, this is complete enough for an agent to call it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% for the single 'id' parameter, so the baseline is 3. The description adds meaningful context by explaining that ids come from evaluate_display, giving concrete examples, and noting that the 'light:' prefix is optional.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states that the tool returns one light's definition from the colregs data package, listing the exact fields included. It is distinct from sibling tools because it ties ids to evaluate_display, but it does not explicitly name or contrast the other sibling tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit scope and exclusions: COLREGS Part C lights only, international text only, lights at night only, and no day shapes/sound signals/Part B rules. It also clarifies the id provenance via evaluate_display, helping an agent decide when this tool is appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• rule_textA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The annotations already establish read-only and idempotent behavior, but the description adds substantial behavioral context: the source file, retrieval date, no-paraphrasing guarantee, pre-release data quality, and version instability. This goes well beyond what annotations alone convey.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose, followed by cite syntax and then important limitations. It is longer than strictly necessary for a one-parameter tool, but every sentence contributes useful safety or provenance information, so the length is justified.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple single-parameter read-only tool, the description is complete: it defines the parameter format, scope, source, caveats, and versioning behavior. Even without an output schema, it explains what the response contains (verbatim text plus source and retrieval date).

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already has 100% coverage for the 'cite' parameter with examples, so the baseline is 3. The description adds meaningful extras: bare rule numbers return every paragraph of that rule, ranges are accepted, and cites follow the exact format used elsewhere in the system. This strengthens the parameter semantics beyond the schema's short example.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states that the tool returns the verbatim text of a COLREGS paragraph from a specific data source, addressed by cite. The verb 'retrieve' is implied through 'the verbatim text ... by its cite,' and the detailed cite syntax distinguishes it from sibling tools like light or evaluate_display.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives strong context for when to use the tool: whenever verbatim paragraph text is needed by a COLREGS cite, and it explicitly lists coverage exclusions (no day shapes, no sound signals, no Part B rules). It does not explicitly name alternative tools or say 'use X instead,' but the scope and limitations are clear enough to guide selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• applied_entriesA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint and idempotentHint, and the description adds substantial non-obvious behavior: output ordering, verbatim provenance from the data package, coverage limitations, pre-release data status, and a stability warning that the surface may change without notice. This goes well beyond what annotations provide.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is dense but well organized: purpose first, then the key alternative, then scope limitations, then data-provenance and stability caveats. Every sentence adds necessary information, and the most decision-relevant content is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complex nested input and no output schema, the description is unusually complete: it specifies the output shape ({id, cite} pairs in data order), the exact coverage boundaries, the source data version, the preview instability, and the fact that entries are verbatim rather than paraphrased. Nothing essential for invoking the tool correctly is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The single top-level facts parameter has no inline description in this tool, and schema description coverage is 0%. However, the description says 'Same input as evaluate_display', linking the agent to the sibling schema, and frames the input as a fact record. The nested schema itself documents each fact property in detail, so an agent can construct valid input despite the top-level cross-reference.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: it returns COLREGS applicability entries that hold for a fact record, as {id, cite} pairs in data order. It also clearly distinguishes itself from evaluate_display by framing this as the cheaper 'which paragraphs speak to this vessel' question, so an agent can tell the tools apart.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly tells the agent when to use evaluate_display instead and what this tool is for: 'Use evaluate_display for the lights; this is the cheaper question'. It also gives hard exclusions via the coverage statement: no day shapes, no sound signals, no Part B steering rules, and not for navigation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.