Server Quality Checklist

Tool Scores

• compare_trainsB
  3.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the safety profile is covered. The description adds the behavioral note that it does not provide a subjective recommendation, which is useful context not present in annotations. However, it doesn't describe what happens to the input (e.g., whether it returns a new sorted list or modifies in place), though given readOnly and idempotent hints, this is largely implied. The description adds value but not deeply.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no wasted words. The intent is front-loaded, and the clarifying statement about not making subjective recommendations is succinct and adds value without bloat. This is an appropriately concise description.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 10 parameters (3 required) and schema coverage is only 20%, the description is far from complete. It doesn't explain what input structure is expected (though it says 'structured direct-train candidates'), how the tool integrates with siblings like search_trains, or the meaning of most parameters. While an output schema exists, the input semantics and usage context are inadequately described for an agent to call it correctly without additional information.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is only 20%, meaning the description must compensate for the many undocumented parameters. The description explains the sort criteria (departure, arrival, duration, total fare) which maps to the sortBy enum, but it fails to explain other critical parameters like fromStationId, toStationId, date, limit, offset, operators, serviceTypes, and time filters. With 10 parameters and such low coverage, the description does not help an agent understand how to construct a valid request beyond the sort field.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific action: sorting structured direct-train candidates by departure, arrival, duration, or total fare. It distinguishes itself from recommendation tools by explicitly noting it does not make a subjective recommendation. However, it doesn't clarify whether the tool expects a pre-fetched list or fetches its own candidates, leaving some ambiguity about its exact role relative to search tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies that the tool should be used when you have structured direct-train candidates and want to sort them, but it does not explicitly state when to use it versus alternatives like search_trains or search_journeys. It would benefit from a note like 'After search_trains returns candidates, use this to sort them' or an explicit exclusion of other tools. The note about not making a subjective recommendation gives a hint of what it does not do, but not when to choose it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_journeysA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already cover read-only, idempotent, open-world, and non-destructive behavior. The description adds a valuable behavioral detail beyond those: the Shinkansen-first provider reports journey search as unsupported instead of incorrectly returning direct trains. This helps agents interpret empty or error responses correctly.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences deliver the core usage rule and a provider-specific caveat with no filler. The most important guidance is front-loaded, making the description easy to scan and act on.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The selection context is strong and the output schema covers return values, but the tool has 8 parameters with only 25% schema coverage. The description does not compensate for that gap, leaving several parameter semantics unexplained for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is only 25%, covering just fromStationId and toStationId. The description provides no explanation of operators, serviceTypes, includeNonShinkansen, departureAfter, or departureBefore, so an agent has little guidance for correctly setting most parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific, actionable rule: use this tool for routes that may include transfers, not for an individual train. This clearly separates it from search_trains and other sibling tools. The provider caveat reinforces the tool's unique role rather than blurring it.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives explicit when-to-use guidance (routes with transfers) and an explicit when-not-to-use boundary (not for an individual train). It does not name search_trains as the alternative, but the exclusion is strong enough that an agent can infer the correct routing.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_availabilityA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the agent knows it's a safe, non-mutating read. The description adds value by disclosing that the provider returns an explicit unsupported status and never fabricates inventory, which is a critical behavioral detail not covered by annotations. This provides meaningful context beyond the structured fields.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no filler. The key scope ('Use only to check...') is front-loaded, followed by a single behavioral note. Every word earns its place, and the structure is immediately scannable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple read-only check with an output schema (present but not shown), the description covers the core purpose, the main behavioral nuance (unsupported status), and safety via annotations. It does not describe error conditions or validate input requirements, but given the output schema exists and the tool's simplicity, what is provided is adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 50%: fromStationId and toStationId have descriptions ('Canonical ID returned by search_stations.'), while trainId and date have none. The description mentions 'exact train and station pair' but does not elaborate on how to obtain trainId or the date format, nor does it reference train identifiers from any sibling tool. Since coverage is low (<80%), the description should compensate but does not, leaving two parameters poorly documented.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Use only to check whether the configured provider exposes seat inventory for an exact train and station pair', which names the verb (check), the resource (seat inventory), and the precise scope (exact train and station pair). This clearly separates it from sibling tools like get_train_details or search_journeys without ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'Use only' explicitly restricts the tool to this specific check, and the description states that the default provider may return an unsupported status. However, it does not name alternative tools for broader journey planning or explicitly state when NOT to use it, though the restriction is implicit. This is clear context but lacks named alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_stationsA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, openWorldHint=true, idempotentHint=true, and destructiveHint=false. The description adds behavioral context by stating it 'returns candidates' and explicitly says it does not silently resolve ambiguous inputs—this tells the agent to expect multiple results for ambiguous queries, which is not captured in annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with zero waste: the first delivers the usage directive, the second describes behavior. It is front-loaded with the most important instruction and stays compact.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple 2-parameter schema, annotations that cover safety, and an existing output schema (which defines return values), the description covers the essential ambiguity-handling behavior. It does not explain how to use the returned candidates (e.g., passing a station ID to search_trains), but that is adequately implied by the 'use before search_trains' directive. This is complete enough for an agent to invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It explains the query parameter implicitly as a station name in Japanese, English, or romanized form, but it never mentions the 'limit' parameter at all. Since limit is optional with a default, the omission is less critical, but for a tool with only two parameters, the description should clarify both to fully address parameter semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns station name candidates for Japanese, English, and romanized queries, and explicitly distinguishes it from the sibling search_trains by positioning it as a pre-step when no station ID is known. The verb 'Returns' and resource 'Japanese railway stations' give a precise purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  'Use this before search_trains whenever no canonical station ID is known' gives an explicit when-to-use directive, and the note about not silently resolving ambiguous inputs (e.g., Osaka, Fukuoka) further clarifies the appropriate context. However, it does not explicitly state when to avoid this tool beyond 'when ID is known', which is implied but not explicitly framed as an exclusion.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_trainsA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already cover read-only, idempotent, non-destructive behavior, so the description adds value with non-trivial behavioral detail: filters are applied before pagination and only direct services are returned. No contradiction with the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences contain the prerequisite, core scope, required input, and filter/pagination behavior with no filler. The most important usage constraint is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is nearly complete for a read-only search tool: it covers prerequisites, scope, required date, optional filters, and filter-pagination ordering, while the output schema covers return details. It could be slightly stronger by naming the sibling for transfer journeys.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With schema description coverage at only 22%, the description must compensate. It groups optional filters into 'time, service, and operator' and notes they apply before pagination, but it does not map them to departureAfter/departureBefore, serviceTypes, and operators or explain their value semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: 'Searches direct Shinkansen services only, never transfer journeys.' This clearly identifies the tool's scope and semantically differentiates it from transfer-search siblings such as search_journeys.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives explicit sequencing ('Use after resolving both station IDs') and a clear exclusion ('never transfer journeys'), plus a required date. It does not name an alternative tool for transfer searches, so it falls just short of fully explicit sibling routing.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_train_detailsA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, openWorldHint, idempotentHint, and non-destructive behavior, covering the safety profile. The description adds the 'ordered stops' detail, which hints at the response structure, but doesn't disclose additional side effects or edge cases. This is adequate given the strong annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, each earning its place. The primary usage instruction is front-loaded, and the warning about manual construction is a concise, valuable addition. No fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter tool with an output schema available, the description covers the essential usage (source of ID, what to retrieve). An agent has everything needed to call it correctly without external context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0% and the schema only has minLength. The description compensates by stating the trainId is opaque and must come from search_trains, not constructed manually. This adds crucial semantic meaning that the raw schema lacks.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('retrieve') and the specific resource ('that service and its ordered stops'), and explicitly ties it to the opaque trainId from search_trains. This distinguishes it from sibling tools like search_trains or compare_trains without ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives explicit usage context: use with the opaque trainId returned by search_trains, and warns not to construct IDs manually. It doesn't explicitly rule out alternatives, but the instruction is clear enough for an agent to know when to invoke it versus other tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_provider_statusA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already cover read-only, idempotent, and non-destructive behavior. The description adds value by explaining that it shows configuration state and the reasons for unavailable capabilities, which is beyond what annotations provide. No contradiction.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that places the usage guidance first and includes no filler. Every phrase contributes essential context.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  An output schema exists, so return values are already structured. The description covers the tool's purpose, when to use it, and what it reveals, which is complete for a zero-parameter tool with annotations covering safety aspects.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the description carries no burden to explain parameter meaning. Per the baseline for 0-parameter tools, a score of 4 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'see' and resource 'rail provider status', and explicitly describes what the tool reveals: which read-only capabilities are configured and why disabled capabilities are unavailable. This clearly differentiates it from sibling tools that handle live train queries.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives an explicit usage condition: 'Use before live train queries.' This tells the agent when to invoke it. However, it does not name alternative tools or explicitly state when not to use it, so it falls short of a full 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.