Server Quality Checklist

Tool Scores

• get-directionsA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate readOnlyHint and openWorldHint. Description adds return value details (distance, duration, steps, GeoJSON) but does not disclose rate limits, authentication needs, or potential variability across transport modes.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences, front-loaded with purpose. Every sentence adds value with no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description adequately describes return values. Could mention coordinate system assumptions or global coverage limitations implied by openWorldHint, but not critical for basic usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters. The description reiterates that directions are between two points but adds no new meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the tool gets directions between two points, including distance, duration, and turn-by-turn steps, and identifies the output as GeoJSON LineString. It distinguishes itself from siblings like get-isochrone which provides isochrones.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool vs alternatives like get-isochrone or score-venues. Does not mention prerequisites or when not to use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get-isochroneA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description mentions the tool returns a 'GeoJSON polygon', adding value beyond the annotations, which already indicate readOnlyHint and openWorldHint. However, it does not disclose further behavioral traits (e.g., underlying data source, rate limits, or computational cost). The annotations cover safety, so the description minimally supplements transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description consists of two concise sentences with no fluff. It front-loads the core purpose and uses plain language. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the moderate complexity (4 required parameters, no nested objects, no output schema), the description is sufficiently complete. It explains the output format (GeoJSON polygon) and the use case. An output schema is not present, so the description partially compensates. It could mention default routing profile or data source, but overall it is adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema provides 100% coverage with descriptions for all four parameters. The tool description does not add any additional meaning beyond what the schema already states. According to the rubric, high schema coverage yields a baseline of 3, and no extra value is provided, so the score is 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Get a reachability polygon showing everywhere reachable from a point within a given travel time.' This is a specific verb and resource, and it distinguishes the tool from siblings like get-directions and search-venues, which serve different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description includes a general use case ('useful for understanding how far someone can travel') but does not explicitly state when to use this tool versus alternatives. It lacks exclusion criteria or context for selecting this tool over siblings, which limits the agent's decision-making.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• score-venuesA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description mentions computing travel times and ranking by fairness strategy, consistent with the readOnlyHint annotation (no mutation). No additional behavioral traits (e.g., unreachable venues handling, caching, or output details) are disclosed beyond the annotations, but annotations already convey non-destructive nature.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise, consisting of three sentences that front-load the main action, explain the computation, and provide usage guidance. No unnecessary words or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description lacks information about the output format. Given no output schema, it is critical to specify what the tool returns (e.g., ranked list with scores, travel times). This omission hinders the agent's understanding of how to use the result.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, so each parameter's purpose is clear from the schema alone. The description adds minimal extra semantic value beyond reiterating the function; it does not elaborate on fairness strategies or transport mode options beyond schema definitions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: scoring candidate venues by travel time fairness for multiple participants. It specifies the action (score), resource (venues), and context (travel time fairness), and distinguishes it from the sibling 'search-venues' by indicating it is used after venues are gathered.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description advises the AI to suggest venues from its own knowledge or via 'search-venues' before using this tool, providing clear usage context. While it does not explicitly state when not to use or list alternative tools, it implies the appropriate workflow.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search-venuesA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate readOnlyHint=true and openWorldHint=true. The description adds that it uses OSM data and returns specific fields, but does not disclose rate limits, data freshness, or other behavioral traits. With annotations covering safety, the description adds limited behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences: first states action and data source, second lists return fields, third gives usage guideline. No redundant information, well-structured and front-loaded. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a search tool with 4 parameters and no output schema, the description covers purpose, data source, return fields, and usage guidance. It lacks mention of potential limitations (e.g., OSM data staleness) or explanation of venue_types enum, but the schema fills that gap. Fairly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, so each parameter is documented. The description adds examples ('pubs, cafes, restaurants, parks, etc.') and context like 'near a location', but does not significantly add meaning beyond the schema. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches for venues (pubs, cafes, etc.) near a location using OpenStreetMap data, and lists the return fields. This distinguishes it from sibling tools like get-directions or get-isochrone, which serve different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description includes a guideline: 'Use this when you need comprehensive local venue data that may not be in your training knowledge.' This implies when to use, though it does not explicitly state when not to use or name alternatives. Given siblings are distinct, the guidance is sufficient.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• store-routing-credentialsA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations indicate idempotentHint=true and readOnlyHint=false, consistent with storage. Description adds workflow context (when to call and effect on future calls) beyond annotations. No contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, no wasted words, front-loaded with key action and context. Every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple store operation with two well-described parameters, the description provides complete workflow guidance and outcome. No output schema needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. Description adds only minimal context that parameters come from a payment_required response, not enhancing the schema definitions significantly.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Store L402 payment credentials') and the resource ('macaroon + preimage'). It distinguishes from sibling tools by explaining that these credentials enable authentication for routing calls.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says to call after the user has paid a Lightning invoice from a payment_required response. Also notes that subsequent routing calls will authenticate automatically. No explicit when-not-to-use, but the context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.