Server Quality Checklist

Tool Scores

• flight_statusA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses that the tool returns scheduled vs estimated times, delay, and cancellation info, implying a read-only query. However, it does not mention any potential side effects, authorization needs, or response format details, leaving some behavioral aspects unclear.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise, using a single purpose sentence followed by parameter descriptions. It is front-loaded with the core action, and every sentence adds value without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite no output schema, the description hints at return content (times, delay, cancellation). Combined with clear parameter descriptions and a specific purpose, the description is sufficiently complete for an agent to invoke the tool, though explicit output details would improve it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Given 0% schema description coverage, the description effectively adds meaning to all three parameters: carrier_code (IATA, example 'AF'), flight_number (without carrier, example '1234'), and departure_date (ISO format 'YYYY-MM-DD'). This exceeds the raw schema and provides actionable guidance.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the purpose: 'Get the status of a flight' with specifics on what is included (scheduled vs estimated times, delay, cancellation). It distinguishes from sibling tools like search_flights (searching) and predict_delay (predicting) by focusing on current status retrieval. However, it does not explicitly differentiate from all siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives is provided. The description lacks context on prerequisites or when not to use it, such as for historical data or predictions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• predict_delayA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so the description carries the full burden. It does not disclose behavioral traits such as whether the tool is read-only, rate limits, authentication needs, or what happens on invalid input. The description only covers basic purpose and parameters.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficient with a clear purpose sentence followed by a structured 'Args' list. No redundant text, and the most important information comes first.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the tool's purpose and all parameters, but lacks any information about the return value or output format. Since there is no output schema, the description should have mentioned what the prediction result looks like (e.g., probability value). Also missing error handling context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Despite 0% schema description coverage, the description adds value by explaining each parameter with examples (e.g., origin 'NCE', departure_datetime format 'YYYY-MM-DDTHH:MM') and clarifying codes. This helps agents understand required formats beyond schema titles.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool predicts the probability that a flight is delayed by more than 2 hours. It uses a specific verb 'predict', identifies the resource 'flight delay probability', and distinguishes from siblings like 'flight_status' which likely gives current status rather than prediction.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus alternatives. The description implies it's for prediction, but does not mention exclusions or recommend other tools for current status or actions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• handle_disruptionA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description must cover behavior. It lists the steps (status check, assessment, rebooking, policy, message) and notes the scenario parameter is mock-only, but does not clarify if actions are immutable, require permissions, or produce side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with a clear opening on purpose followed by parameter details. Every sentence adds value without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a multi-step tool with no output schema, the description omits what the tool returns (beyond 'grounded passenger message'), error conditions, or idempotency. This gap hinders full understanding of tool behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Despite 0% schema description coverage, the 'Args' section in the description provides thorough explanations for all parameters: carrier_code format, flight_number usage, departure_date ISO format, and the scenario mock options. It also specifies required parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: running a full disruption-handling agent with multiple steps. It distinguishes itself from sibling tools like flight_status and search_flights by bundling those tasks into one orchestration.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when a comprehensive disruption workflow is needed, but it does not explicitly state when to use this tool versus individual sibling tools, nor does it provide when-not-to-use conditions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_flightsA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It specifies constraints (non-stop, one adult) but does not disclose whether it is a live search, whether it modifies data, or the format of return results. Additional context on side effects or access patterns would be beneficial.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Description is concise with a clear header and a straightforward list of args. Every sentence adds value; no wasted words. Well-suited for quick comprehension.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the presence of an output schema, the description does not need to detail return values. It covers the essential input semantics and constraints. However, it could mention that the search is for commercial flights (bookable) and that results may vary over time.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It explains all four parameters with examples (IATA codes, ISO date format) and default values for max_results. This adds significant meaning beyond the bare schema properties.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states it searches bookable flights for a route and date, with explicit constraints (non-stop, one adult). It differentiates from sibling tools like flight_status or handle_disruption by focusing on searching, not status or disruption handling.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Implied usage for searching flights, but no explicit guidance on when to use this tool over alternatives like flight_status or predict_delay. No exclusions or use-case boundaries mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• passenger_rightsA
  4.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses the return format ('most relevant policy passages with sources') but does not mention side effects, permissions, or limitations. This is adequate but not thorough.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences front-loaded with purpose and value, followed by a clean Args block. Every sentence serves a purpose with no waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool is a retriever with an output schema, the description adequately covers its function. It mentions sources and relevance, which combined with output schema, provides complete context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description compensates fully. It explains both parameters: question with example, and top_k with default value, adding meaning beyond schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Look up passenger rights and carrier disruption policies' with specific references (EU261, rebooking, baggage, duty of care). It distinguishes from siblings like flight_status and handle_disruption by focusing on policy lookup with citations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use when needing to cite entitlements with sources. However, it does not explicitly state when not to use or compare to alternatives, leaving some guidance unstated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.