Server Quality Checklist

Tool Scores

• get_weatherA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully discloses routing logic, date constraints (1940+ lag, 16-day forecast), and default behavior when dates are omitted. It does not cover rate limits or permissions, but provides enough detail for safe invocation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (routing logic, examples, when to use, returns). It is relatively lengthy due to examples, but front-loaded with purpose. Each part serves a purpose, though some repetition could be trimmed.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is comprehensive for a tool with routing logic and date complexity. It covers all key behaviors, provides examples for different scenarios, and explains the return type. With an output schema existing, further detail is not needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, but the description adds significant value by explaining automatic routing based on date values, showing example calls, and clarifying granularity impact (e.g., ~24x more records for hourly). This exceeds the baseline of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool queries weather over a date range and distinguishes it from sibling tools like 'latest' which handles current conditions. It specifies automatic routing between historical and forecast endpoints.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The 'When to use' section explicitly lists three use cases (time-series queries, multi-day analysis, climate research), but does not mention when to avoid using it or suggest alternatives like 'latest' for current conditions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compare_locationsA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description carries full burden. It details concurrent fan-out via asyncio.gather, independent resolution of each location, error handling (failures yield error fields without affecting others), and caching behavior. This provides comprehensive insight into the tool's operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (purpose, behavior, examples, when to use, returns). It is informative without being verbose, though slightly more conciseness could be achieved by trimming some repetitive elements.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the output schema exists, the description appropriately focuses on input usage and behavioral details. It covers single-parameter usage thoroughly, explains error behavior, and mentions return structure, making it self-contained for effective agent invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and the schema already describes the 'locations' parameter well. The description adds value by providing concrete examples and clarifying that mixed input shapes (IDs, names, postcodes) are accepted, which enhances usability beyond the schema alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Compare current weather across multiple Australian locations in one call.' It specifies the verb (compare), resource (current weather), and scope (multiple Australian locations). This clearly distinguishes it from sibling tools like 'get_weather' which handles a single location.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The 'When to use' section provides clear use cases such as comparing weather in several cities, building multi-region dashboards, or planning holidays. It does not explicitly list alternatives, but the context strongly implies when to use this tool versus single-location tools like 'get_weather'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• describe_locationA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It discloses that input can be various shapes (curated IDs, place names, states, coordinates), explains the behavior of the returned 'id' (None for non-curated, snake_case for curated), and mentions case-insensitivity. It lacks an explicit read-only declaration but implies a safe lookup.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and concise: it starts with a clear purpose, then covers input flexibility, return behavior, examples, usage guidance, and return fields. Every sentence adds value without redundancy, achieving a balance between completeness and brevity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that an output schema exists, the description appropriately covers the key return fields (LocationDetail with id, name, state, lat/lng, timezone, elevation, BOM station ID, Open-Meteo URL, attribution). It also addresses non-curated vs. curated behavior and provides usage context, making it complete for the tool's purpose.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% (the 'location' parameter is well-documented). The description adds significant value beyond the schema by providing multiple examples illustrating different input shapes, explaining the relationship between input and output (especially the 'id' behavior), and clarifying case-insensitivity.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states 'Return metadata for an Australian location' with a specific verb and resource, listing fields like name, lat/lng, timezone, elevation, and BOM station ID. It clearly distinguishes itself from sibling tools (e.g., get_weather, search_locations) by focusing on metadata lookup, not weather or search.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description includes a 'When to use:' section with three concrete scenarios: before get_weather, for BOM cross-reference, and to verify ambiguous input. While it doesn't explicitly state when not to use, the positive guidance is clear and sufficient for an agent to recognize appropriate contexts.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• latestA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully discloses behavior: 15-minute cache TTL, wrapper around Open-Meteo, latency, response fields like 'location_resolution', and attribution. No contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with sections for purpose, implementation, examples, usage, and returns. It is front-loaded and every sentence is informative, though slightly lengthy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the low complexity (1 parameter, output schema exists), the description is comprehensive: covers caching, resolution, use cases, and return fields. No gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds value with multiple examples and explains how inputs are interpreted (curated, geocoded, etc.), enhancing understanding beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it returns current weather for any Australian location, with a specific verb and resource. It differentiates from siblings like 'get_weather' and 'air_quality' by focusing on current conditions and fast cache.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The 'When to use' section provides clear canonical use cases and practical scenarios. It does not explicitly state when not to use or name alternatives among siblings, but the examples and context effectively guide usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_locationsA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses fuzzy-matching behavior, coverage of curated set, resolution for non-curated places, and return format. It could note that it's read-only, but the examples imply no side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-organized with clear sections, bullet points, and examples. Every sentence is informative and earns its place, avoiding unnecessary fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the output schema exists, the description appropriately summarizes return format (List of LocationSummary with id, name, state, description). It integrates well with sibling tools and provides enough context for an agent to decide when to use this tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds value by providing examples (e.g., 'tropical north') and clarifying matching behavior (case-insensitive, matches IDs, names, state codes), which goes beyond the schema's description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool does 'Fuzzy-search the 45 curated Australian locations.' It specifies the verb and resource accurately and distinguishes from siblings like list_curated (which returns the full set) and describe_location.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description includes a 'When to use' section with three specific use cases: discovering location ID, finding all locations in a state, and verifying if a place is curated. It also mentions alternatives for outside the curated set, referencing list_curated() and geocoding.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• air_qualityA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description fully discloses the data source (Open-Meteo, Copernicus CAMS), units (µg/m³), and output structure (pollutants, AQI scales, plain-English labels). It also notes seasonal relevance and coordinates. No contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a clear opening, source note, examples, use cases, and return summary. It is concise without unnecessary repetition, and key information is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter) and the presence of an output schema, the description is complete. It covers all aspects: purpose, usage, parameter, output, and context. No gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with a single 'location' parameter. The description adds significant meaning beyond the schema by providing multiple examples (city names, postcodes, coordinates) and explaining that it accepts curated IDs, place names, states, and lat,lng. This enriches semantic understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states it returns current air-quality readings for Australian locations, listing specific pollutants and AQI indices. It distinguishes from sibling tools like get_weather and compare_locations by focusing solely on air quality.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The 'When to use' section provides concrete examples such as checking air quality before running, during bushfire season, for asthma planning, and long-term monitoring. It lacks explicit 'when not to use' or alternatives, but the guidance is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_curatedA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description fully discloses behavior: fast-path lookup, inclusion in search_locations, and the YAML-based update process. It also specifies the return format (sorted list of 45 IDs).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections and bullet points, making it easy to read. It is somewhat lengthy but each sentence adds value. Minor redundancy could be trimmed, but overall it is efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the absence of parameters and presence of an output schema, the description covers purpose, usage, return value, and even internal maintenance. It is complete for the complexity of this simple list tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the schema is trivially covered. The description adds no parameter details, which is acceptable as baseline for no parameters. The description does not need to add meaning beyond the empty schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists 45 curated Australian location IDs, with a specific verb and resource. It distinguishes from sibling tools like search_locations by noting that non-curated locations work via a geocoder, while curated entries get fast-path lookup.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  An explicit 'When to use' section provides three concrete scenarios: enumerating supported locations, building UIs, and planning multi-location dashboards. It also implies when not to use by referencing alternative tools for locations outside the curated set.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.