Server Quality Checklist

Tool Scores

• resolve_airlineA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral transparency. It discloses that the tool accepts partial names or exact codes and returns matches, implying a lookup operation. However, it doesn't mention whether the tool is read-only, if there are rate limits, or what happens on no match, which are gaps for a resolution tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with two sentences and parameter details. It front-loads the main purpose before listing args. The arg descriptions are clear and efficient, though the full description is about 90 words, which is reasonable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 2 params, 0% coverage, no output schema, and moderate complexity, the description covers purpose and parameter constraints. However, missing output details, error behavior, and use cases limits completeness. It's adequate but not thorough.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It does explain keyword as 'Airline name or partial name... or exact IATA code' and max_results with range and default. This adds meaning beyond the schema's types, but doesn't describe return format or error handling, and max_results semantics could be clearer (e.g., what happens if fewer matches exist). Baseline with 0% coverage is 3, and description meets that minimum.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: to look up airline IATA codes by airline name or exact IATA code. It specifies the resource (airline codes) and the action (look up by name or code), and differentiates from siblings like search_flights and get_flight_status.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use the tool (resolving airline names/codes) and usage tips like minimum 2 characters. However, it does not explicitly state when not to use it or compare to siblings like resolve_airport, though sibling names imply they handle different domains.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_flight_statusA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It discloses that the tool returns current status (scheduled/active/landed/cancelled/etc.) and optional live position, which is helpful. However, it does not explicitly state that the operation is read-only (though implied by 'get'), nor does it cover rate limits, authentication needs, data freshness, or error handling. Adequate for a simple tool but could be more transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise: one sentence for purpose, a short args section, and a return summary. Every sentence adds value, and the structure is front-loaded with the main action. No redundant phrases.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter, no output schema), the description covers the purpose, parameter meaning, and return values clearly. It does not mention edge cases (e.g., invalid flight code) or data freshness guarantees, but it is largely complete for a straightforward retrieval tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% description coverage, but the description explains the single parameter flight_iata with an example ('AA100') and clarifies the format ('airline code + flight number'). This adds significant meaning beyond the raw schema definition and fully compensates for the lack of schema-level descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The first sentence clearly states the tool's purpose: 'Get the real-time status of a specific flight by its IATA flight code.' This uses a specific verb ('Get') and a well-defined resource ('real-time status of a specific flight'). The tool is distinct from siblings like search_flights (which searches by criteria) and resolve_airport/airline (which resolve codes).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Usage is implied: when you have an IATA flight code and want real-time status. However, there is no explicit guidance on when to use this tool versus alternatives (e.g., search_flights if code is unknown). No when-not conditions are stated, so it barely meets 'implied usage'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• resolve_airportA
  4.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses input constraints (minimum 2 characters, max_results range) and core behavior, but does not explicitly state that it is read-only, what happens on no match, or the output format. The description is adequate but lacks deeper behavioral details that an agent might need.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is succinct with two clear paragraphs. The first sentence front-loads the purpose, followed by precise argument definitions in a standard format. No redundant or unnecessary sentences.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema and no annotations, the description covers the tool's purpose, usage context, and parameter semantics well. The only minor gap is the lack of explicit output structure (e.g., 'returns a list of IATA code strings'), but the verb 'look up' and typical use case imply the result. For a simple lookup tool, it is largely complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must add meaning. It provides rich details for both parameters: keyword examples ('London', 'Heathrow', 'JFK'), minimum length (2 chars), and max_results range (1-20, default 10). This fully compensates for the empty schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('look up') and resource ('IATA airport codes') and specifies the input as 'city or airport name'. It distinguishes itself from siblings (search_flights, resolve_airline) by its specific purpose and the explicit guidance to use it before search_flights.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly tells when to use the tool: 'Use this before search_flights if you only have a city/airport name rather than a 3-letter IATA code.' This provides clear context and an alternative (direct IATA code usage) without needing additional exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_flightsA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description fully bears the transparency burden. It declares that only current/near-term real-time flights are returned, excludes future/historical flights and prices, and warns about unfiltered calls. This sufficiently discloses the tool's behavioral constraints.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose, followed by key caveats in a clear 'IMPORTANT' block, then a structured 'Args' list. Every sentence adds value, with no unnecessary words. The organization aids quick comprehension.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 5 parameters, no output schema, and no annotations, the description covers scope, limitations, parameter details, usage best practices, and sibling alternatives. It leaves no critical gaps for an agent to misuse the tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must and does compensate by explaining each parameter: origin/destination as 3-letter IATA codes, max_results range/default, airline_name as optional name, and flight_status with enumerated allowed values. The filter requirement is also clarified.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Search current/live flights' and specifies filtering by route, airline, or status. It explicitly differentiates from sibling 'get_flight_status' by noting the scope and directing specific status queries to that tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit when-to-use (current/near-term real-time flights), when-not-to-use (future dates, historical, prices), and recommends the alternative 'get_flight_status' for specific flight status. It also advises providing at least one filter to avoid quota waste.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.