Server Quality Checklist

Tool Scores

• get_forecastC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so the description must carry full behavioral disclosure. It does not mention any behavioral traits such as whether the operation is read-only, any authentication needs, or rate limits. The term 'raw calibrated forecast context' is vague.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very short and includes a clear parameter list. Every sentence is functional, but it could be more structured with headings or bullet points.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has an output schema (not shown), the description need not explain return values. However, for a simple tool with one parameter, it is minimally complete but lacks guidance on error conditions or alternative tools.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% description coverage, but the description compensates by listing the allowed cities (nyc, chicago, denver, miami, la). However, it does not explain the meaning of 'city' beyond these values or provide formatting details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Get') and resource ('raw calibrated forecast context'), and it is distinct from sibling tools which focus on signals or observations. However, it does not explicitly differentiate itself from siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives like get_weather_signals or get_station_observation. The description only lists supported cities without explaining the tool's role.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_station_observationB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description must carry behavioral disclosure. It mentions the tool provides the latest METAR observation and lists allowed cities, but lacks details on error handling, cache behavior, or output structure. The output schema exists but is not described.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very concise with two sentences, including a structured Args section. Every sentence is necessary, though front-loading could be improved by placing the allowed cities list earlier.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with one parameter and an existing output schema, the description is mostly complete. It covers the purpose, parameter, and allowed values. Missing are any notes on return format or potential errors, but these are mitigated by the output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaning by enumerating allowed city values (nyc, chicago, denver, miami, la) which the input schema does not provide (no enum). Since schema coverage is 0%, this partially compensates, but does not explain format or constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it retrieves the latest METAR observation for a single city, which distinguishes it from sibling tools like get_all_signals or get_forecast. However, it does not explicitly contrast with these alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description lists valid city values but provides no guidance on when to use this tool instead of siblings like get_forecast or get_all_signals. No exclusion criteria or context are given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_citiesA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description is the sole source of behavioral information. It only describes the action as 'List', implying read-only, but offers no details on side effects, data source, or reliability. More context is needed for safe invocation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence of eight words, front-loaded with the verb. Every word is essential, and there is no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has no parameters and an output schema exists, so the description is minimally adequate. However, it does not hint at the structure or size of the output, nor any filtering capabilities. For a trivial tool, this is sufficient but not thorough.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has zero parameters, so schema description coverage is 100%. The description does not need to explain parameters. Baseline 4 is appropriate as no additional semantic is required.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists supported cities, settlement stations, and calibration parameters. The verb 'List' combined with specific resources differentiates it from sibling tools that deal with signals, forecasts, and observations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. There is no mention of prerequisites, context, or conditions for invocation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_all_signalsA
  4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full responsibility for behavioral disclosure. It describes the action as a 'full scan' but does not mention any side effects, potential rate limits, authorization requirements, or whether the scan is resource-intensive. The agent has no information about safety or performance implications, which is a significant gap for a potentially expensive operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description consists of a single sentence that clearly and efficiently conveys the tool's purpose. Every word is meaningful, with no redundancy or filler. It is appropriately sized for its simplicity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no parameters and an output schema exists (though not shown in this evaluation), the description is largely complete. It states the tool scans all cities and ranks signals. However, it could benefit from noting that this is a broad, potentially expensive operation, and that the output schema details the results. Overall adequate but slightly lacking in operational context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has no parameters, and schema coverage is 100% (empty schema). The description adds no parameter-level information because none is needed. With zero parameters, the baseline for this dimension is 4, and the description does not detract from it.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it runs a full scan across all supported cities and ranks top weather-market signals. It specifies the verb ('run a full scan'), resource ('signals'), and scope ('all supported cities'), distinguishing it from siblings like get_weather_signals which likely filter by city or other criteria.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implicitly indicates when to use this tool: for a comprehensive scan across all cities. However, it does not explicitly state when not to use it or mention alternatives such as get_weather_signals for more targeted queries. The context of sibling tools provides some guidance, but the description lacks explicit exclusions or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_weather_signalsA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description carries the burden. It implies a read operation ('Get') and lists city options but does not explicitly state side effects, permissions, or rate limits. However, the simple retrieval nature and presence of output schema mitigate missing details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise (two sentences), front-loads the purpose, and includes the parameter specification without redundancy. Every word adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With a single parameter and an output schema, the description covers the essential purpose and allowed values. It does not explain return structure (handled by output schema) or any prerequisites, but is sufficient for basic usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% description coverage and no enum, but the description adds the allowed values 'nyc, chicago, denver, miami, la' for the city parameter, fully compensating for the schema gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description specifies the verb 'Get' and the resource 'calibrated edge signals for one city's Kalshi weather markets', clearly distinguishing it from siblings like get_all_signals (broader) and get_forecast (different resource).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly lists the allowed city values and states 'one city', but does not explicitly guide when to use this tool vs alternatives like get_all_signals for multiple cities.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.