Server Quality Checklist

Tool Scores

• find_routeA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses that the tool computes a route and that it depends on resolve_operational_point, but it doesn't state any side effects, read-only nature, or error behaviors. However, since the tool is a pure computation with no destructive actions implied, this is adequate but not rich. The dependency advice adds some behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no filler. The purpose is front-loaded, and the usage guidance is placed immediately after. Every word earns its place, making it highly scannable for an agent.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having an output schema (which explains return values), the tool has many parameters (objective, allowed_gauges_mm, avoid_section_ids, detail, waypoints_for_weather) that are entirely undocumented in the description. The description only clarifies the op_id parameters indirectly via the mention of resolve_operational_point. Given the tool's complexity and zero schema coverage, this is a significant gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description fails to compensate. It mentions 'op_id values' but does not explain that from_op_id and to_op_id are required, nor does it explain objective, allowed_gauges_mm, avoid_section_ids, detail, or waypoints_for_weather. An agent cannot infer what these parameters mean from the description alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Compute'), the resource ('a route between two operational points'), and the context ('over the ERA RINF infrastructure graph'). This distinguishes it from siblings like resolve_operational_point (resolves points) and check_train_compatibility (checks compatibility), making the tool's purpose immediately clear.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly instructs the caller to 'Call resolve_operational_point first' and to call this tool 'before check_train_compatibility or assess_weather_risk' because both need the route. This provides direct ordering and dependency guidance, leaving no ambiguity about when to use it relative to siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_train_compatibilityA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses the key constraint that exactly one of route or section_ids must be provided, which is valuable. However, it does not mention whether the operation is read-only, any side effects, or behavior around strict_unknown and im_max_train_length_m. It does not contradict annotations (none exist), but it is missing some behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences, front-loaded with the tool's main purpose and then usage. Every word earns its place, with no fluff or repetition. Strong structure.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity and the presence of an output schema (so return format need not be explained), the description covers the primary usage pattern adequately. It misses some parameter details like strict_unknown and im_max_train_length_m, but these are defined in the schema. Overall, it is complete enough for an agent to call the tool correctly in the main intent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 0% in the description text, so the description must compensate. It explains the relationship and requirement between route and section_ids, adding meaning beyond the bare schema. However, it does not explain train, strict_unknown, or im_max_train_length_m, leaving significant gaps. The partial compensation merits a baseline 3 rather than lower.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool checks train-route compatibility across gauge, axle load, clearance, and length. It also distinguishes itself from find_route by referencing it as a prerequisite, though it does not explicitly differentiate from siblings like assess_weather_risk. The purpose is specific and understandable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly instructs to call find_route first and pass its result, or alternatively pass section_ids for a specific segment. It clearly states that exactly one of the two is required, providing clear when-to-use and when-to-use-alternative guidance. This is exemplary usage direction.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• resolve_operational_pointA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the disclosure burden. It adds useful behavioral context: diacritics are ignored, and an empty list is a valid 'not found' result. However, it does not disclose response structure, sorting, or any other side effects. Given the absence of annotations, it provides some value but not comprehensive coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded with the purpose, then immediately gives usage context ('Call this first') and a critical edge-case behavior. Every sentence earns its place with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a name-resolution tool with an output schema (indicated by 'Has output schema: true'), the description covers the core purpose, usage flow, and key edge case. It does not detail how to handle multiple matches, but that is likely covered by the output schema and is not a critical omission for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema itself contains detailed descriptions for all properties (query, limit, types, countries), including the diacritics normalization. The description echoes 'by name' and 'diacritics ignored' but adds nothing materially new beyond the schema. Since the schema already documents parameters well, baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Find'), resource ('operational points'), and specifics ('by name in Poland or Czechia'). It also explicitly connects to the sibling tool find_route by stating it produces an op_id for that purpose, distinguishing it from route planning and other tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Call this first' to resolve a place name to an op_id for find_route, which is clear when-to-use guidance. It also clarifies that an empty candidate list is a valid outcome, not an error. However, it doesn't explicitly state when NOT to use it or compare it to other alternatives, so it lacks full exclusion guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• assess_weather_riskA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must carry the behavioral burden. It discloses the dependency chain and the exclusive-or input pattern, but it does not state whether the call is read-only, how default thresholds are applied, or what happens to the structured output.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences with no filler: purpose first, then prerequisites and input constraint. The 'same as check_train_compatibility' reference efficiently avoids repeating sibling documentation.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with a nested input schema and existing output schema, the description covers the critical cross-tool dependency and the mutually exclusive input requirement. It could add a sentence about weather_points construction or threshold defaults, but the schema already documents those fields.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaning for route/section_ids (exclusive-or, prerequisite via find_route) beyond the schema. However, schema description coverage is reported as 0%, and train, weather_points, and thresholds_override semantics are left to the schema alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The first sentence states a specific operation: turning weather forecasts at points along a route into a structured risk assessment. This clearly separates the tool from the route-finding and compatibility siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly orders prerequisites: call find_route first for route/section_ids, call the OpenWeather weather tool for forecasts at waypoints, then this tool. It also states the exact-one-of route/section_ids requirement, mirroring check_train_compatibility.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• plan_tractionA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description usefully reveals that the assignment is greedy, minimizes changes, and treats uncovered runs as non-errors. However, it mentions 'unmatched signalling' as a cause of uncovered runs without noting that this depends on require_signalling_match, which defaults to false in the schema. With no annotations present, this ambiguity matters and weakens transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three tightly written sentences, starting with the core purpose, then moving to prerequisites, then edge-case behavior. Every sentence adds necessary information and there is no redundant repetition of schema details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the essential invocation prerequisites and the unusual 'uncovered, not an error' behavior, and an output schema exists for return-shape information. The main gap is not explicitly clarifying the signalling condition and its default, but the nested schema partially compensates for that.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is low at the top level, and the description compensates by explaining the relationship between route, section_ids, fleet_ids, and custom_locomotives. It tells the agent where route comes from, that section_ids follows check_train_compatibility semantics, and where locomotive ids originate. The require_signalling_match parameter is left to the schema, but that field already has a detailed description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The opening sentence clearly states the verb, resource, and goal: 'Assign locomotives from a fleet to a route's runs, greedily minimising changes.' This distinguishes it from sibling tools like find_route and check_train_compatibility, whose purposes are route discovery and compatibility checking rather than assignment.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit operational guidance: call find_route first and pass its route, or pass section_ids like check_train_compatibility, and provide at least one locomotive via fleet_ids or custom_locomotives. It lacks an explicit 'when not to use this tool' or a direct comparison to a sibling alternative, but the invocation workflow is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.