Server Quality Checklist

Tool Scores

• wx_sigmetsB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description bears full responsibility for disclosing behavior. It only states what the tool does and mentions a filter; it does not describe the output format, potential side effects, or any limitations (e.g., meaning of 'active', data source, return structure). This is minimal behavioral transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that front-loads the core action and adds a brief usage tip. Every word contributes, making it highly concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with one optional parameter and no output schema, the description is adequate but sparse. It lacks details on what the returned data looks like, how 'active' is determined, or any limitations. Given the minimal complexity, it achieves a 3 but could benefit from additional context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema fully documents the 'region' parameter with enum and a description. The tool description adds the rationale for using the filter ('reduce noise'), which provides slight added value beyond the schema. Per the baseline for high schema coverage, this scores a 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's primary function with a specific verb ('Get') and specific resources (SIGMETs and AIRMETs). It is distinct from sibling weather tools that focus on METAR, TAF, PIREPs, etc., but does not explicitly name alternatives, so it falls short of a 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides a usage tip ('Filter by region to reduce noise') but no explicit guidance on when to choose this tool over siblings like wx_metar or wx_taf. The context is implied (if you need SIGMET/AIRMET data, use this), but there are no exclusions or alternatives mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• wx_metarA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the transparency burden. It states that both raw METAR and decoded fields (ceiling, visibility, wind, flight category) are returned, which is useful behavioral context. However, it does not disclose potential limitations such as data freshness, units, or behavior with invalid ICAO codes.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that front-loads the core action and result. Every phrase adds relevant information without waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  This is a simple single-parameter retrieval tool with no output schema. The description adequately summarizes the tool's functionality and return value, covering what an agent needs to know to invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already fully documents the 'icao' parameter with examples (100% coverage). The description adds no additional parameter semantics beyond restating 'airport', so the baseline score of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description specifies a clear action ('Get the latest METAR') and a specific resource ('an airport'), and distinguishes this from sibling weather tools (e.g., wx_taf, wx_pireps) by naming the METAR product. The return contents are also stated, making the tool's purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage ('when you need a METAR') but gives no explicit guidance on when to prefer this over related tools like wx_taf or wx_pireps. There is no mention of exclusions or alternatives, leaving the agent to infer the use case.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• wx_pirepsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of disclosure. It mentions the type of reports and geographic scope but does not cover return format, pagination, or error behavior. As a read-only 'Get' operation, some safety is implied, but edge cases are unaddressed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences with the action and resource upfront. It avoids redundancy and every clause serves a purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with two parameters and no output schema, the description sufficiently conveys what the tool does and the nature of the returned data. It does not specify the exact output structure, but the listed report types give a good sense of the payload.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Both parameters are already well-described in the schema (icao, radius_nm with default), and the description adds no additional parameter semantics. With 100% schema coverage, a baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the tool's function with a specific verb ('Get') and resource ('pilot reports'), and it lists the types of data (turbulence, icing, cloud tops), which distinguishes it from sibling weather tools like METAR and TAF.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use the tool (when PIREPs near an airport are needed) and implicitly differentiates it from siblings, but it does not explicitly state exclusions or name alternative tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• wx_preflightA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It discloses the data sources and authority for coordinates, but doesn't describe return format, freshness guarantees, or any limitations. This is adequate but not rich.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no wasted words; the key purpose and input constraint are front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple (2 params, no output schema) and the description covers purpose, data sources, and usage constraint. It falls short of describing the assessment output structure, but this is not critical for a preflight brief.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so parameters are already fully documented. The description reinforces that only 'icao' is required and warns against supplying coordinates, but adds no syntax or format details beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific resource ('VFR preflight weather brief for an airfield') and a specific composite action ('Combines current METAR, TAF, and ICON-D2 NWP forecast'), clearly distinguishing it from sibling single-source tools like wx_metar and wx_taf.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It implicitly defines when to use it (for VFR preflight) and gives explicit input guidance ('Only the ICAO code is required', 'Never supply coordinates yourself'). However, it doesn't explicitly contrast with alternatives or state when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• wx_tafA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It clearly communicates a read-only retrieval operation via 'Get' and 'Returns', and it enumerates the returned content (raw TAF, decoded periods with flight category, wind, visibility, ceiling). It does not mention edge cases like missing TAFs or units, but for a simple fetch tool this is reasonably transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is one sentence, front-loaded with the primary purpose, and every clause adds useful information about the returned data. There is no filler or repetition of the tool name.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter forecast retrieval tool with no output schema, the description sufficiently covers the main return components. It does not specify handling of unavailable TAFs, time horizons, or units, but the core functionality is adequately described given the tool's simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% for the single 'icao' parameter, which is already described as 'ICAO airport code'. The description does not add extra meaning about the expected format, examples, or ICAO rules, so it does not go beyond the schema. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool 'Get the Terminal Area Forecast (TAF) for an airport', identifying a specific verb and resource. It distinguishes itself from sibling weather tools by explicitly naming the TAF product, and the mention of 'raw TAF plus decoded forecast periods' offers a clear scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool (when a Terminal Area Forecast is needed), but it does not explicitly state alternatives or exclusions such as 'for current conditions use wx_metar'. The usage context is inferable from the TAF name and the presence of weather siblings, but no direct guidance is given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• wx_icon_d2A
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full transparency burden. It discloses a key behavioral trait: fallback to lower-resolution models outside ICON-D2 coverage. It also lists output variables and model resolution, but omits details like return format, units, or error behavior. Still, it provides substantial insight beyond the bare tool name.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured: three sentences cover purpose, variables, model coverage, and fallback behavior. Every sentence contributes meaningful information, with the key action front-loaded in the first sentence.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the essential context: coverage area, resolution, forecast horizon, output variables, and fallback behavior. No output schema exists, so the description appropriately communicates what the user receives. It could mention output format or units for full completeness, but the provided details are sufficient for a forecast tool at this complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% for all three parameters (lat, lon, hours), with clear descriptions in the schema. The tool description adds context about what the forecast includes but doesn't elaborate on parameter units or formats beyond the schema. Since the schema does the heavy lifting, a baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Get DWD ICON-D2 NWP forecast for any lat/lon.' It enumerates the specific weather variables (hourly wind, visibility, cloud base, precipitation, CAPE) and model characteristics, distinguishing it from sibling tools like wx_metar or wx_taf which are observational reports.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear context: ICON-D2 covers Central Europe at 2km resolution up to 48h, with a fallback to ICON-EU or ICON-seamless outside coverage. This tells users when to use the tool and what to expect based on location, though it doesn't explicitly name alternative tools for other forecast needs.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• wx_routeA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full behavioral disclosure burden. It reveals the tool's composition, the sampling points along the great-circle route, and the dependency on the station API for coordinate resolution. It does not describe output format or failure modes, but covers key behaviors well.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded: the first sentence states the core purpose, the second details the data sources and spatial scope, and the third gives a critical usage rule. Every sentence earns its place with no redundant wording.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity, four parameters, no annotations, and no output schema, the description provides the essential information: what data is fetched, where it is sampled, and how to handle coordinates. It could have mentioned the output structure or error behavior, but the provided details are sufficient for selecting and invoking the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description adds value by emphasizing that ICAO codes must be passed exactly as given and that coordinates must never be guessed, which is directly relevant to the from_icao and to_icao parameters. The other parameters are already well explained in the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly defines the tool's purpose as creating a route weather brief for a direct flight, listing the exact data sources (METARs, ICON-D2, SIGMETs). It differentiates itself from sibling tools like wx_metar, wx_sigmets, or wx_icon_d2 by combining these into a route-based product.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description states when to use the tool: for a direct flight between two airports, and it gives explicit guidance to never guess coordinates and to pass ICAO codes exactly as given. It does not explicitly name alternatives or state when not to use it, but the route-specific context is clear enough.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• wx_outlookA
  4.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description transparently explains the ensemble models, the NO-GO probability definition, thresholds, and the ICON-EU cross-check. It does not detail output formatting or error behavior, but the operational mechanics are well disclosed for a read-only forecast tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Though dense, the description front-loads the core purpose and every sentence contributes unique operational detail—models, probability metric, cross-check logic, and input modes. It is appropriately sized for the tool's complexity with no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a multi-model tool with 8 parameters, no output schema, and no annotations, the description fully covers purpose, inputs, methodology, and relation to siblings. It sufficiently describes what is reported (probability, deterministic trend, cross-check) for an agent to invoke correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema descriptions already cover 100% of parameters, and the description adds valuable semantics: what counts as NO-GO, how thresholds are interpreted, the daylight-window concept, and the offline airport database for ICAO resolution. This goes well beyond standard field names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific resource and action: 'Longer-range VFR confidence outlook (3–15 days)' and contrasts it with short-range METAR/TAF/ICON-D2. This clearly distinguishes it from siblings and leaves no ambiguity about its function.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly positions the tool as the confidence measure for trip planning that short-range siblings cannot provide, naming METAR/TAF/ICON-D2. It also specifies input alternatives (ICAO or lat/lon) and the 3–15 day horizon, giving clear when-to-use guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.