Server Quality Checklist

Tool Scores

• get_metarA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided. The description mentions what is returned but does not disclose behavioral traits such as read-only nature, rate limits, or error handling. Lacks transparency beyond the obvious fetch operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with purpose. Every word is meaningful; no fluff. Efficiently conveys the tool's function.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple 2-parameter tool with no output schema, the description covers the return fields and optional TAF. However, it omits typical error scenarios (e.g., invalid ICAO) and structure of the decoded report.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%; both parameters are described in the schema (ICAO code format, TAF boolean). The description adds 'Optionally include the TAF forecast' but does not provide additional semantic value beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool gets the current decoded METAR for an airport by ICAO code, listing specific fields returned. It is distinct from sibling tools like get_airport or get_aircraft, which handle non-weather data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for aviation weather retrieval but does not specify when to use this tool over alternatives, nor does it provide exclusions or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_aircraftA
  3.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It only lists output fields and example slugs, with no disclosure about read-only behavior, rate limits, authentication, or any side effects. This is minimal for a simple retrieval tool, but still missing guidance on what happens with invalid slugs.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no redundancy. It front-loads the purpose and includes examples efficiently. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with one parameter and no output schema or nested objects, the description covers the purpose, input examples, and output fields adequately. It does not address error handling for invalid slugs, but given the low complexity, it is nearly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with one parameter (slug). The description adds valuable context with concrete example slugs and lists the output fields, which goes beyond the schema's minimal description. This helps the agent select appropriate slug values.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves aircraft specifications by slug and lists the returned fields (engines, seats, etc.), with specific examples. This distinguishes it from siblings like get_airport or get_metar.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implicitly indicates usage for aircraft spec lookups by slug, but it does not explicitly state when to use this tool versus alternatives or provide any exclusions or prerequisites. The sibling context helps but the description itself lacks direct guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_airportA
  3.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavioral traits. It states input and output but does not mention idempotency, error handling, auth requirements, or response format. For a simple read tool, this is a partial disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence that effectively communicates purpose, method, and output without extraneous words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description adequately covers input and output for a simple retrieval tool. It lacks details on error behavior or response structure, but is mostly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema covers 100% of parameters, and the description adds meaningful context: it specifies the parameter is a 4-letter ICAO code and provides an example (KJFK), enriching the schema's description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action 'Get airport information' and the resource 'airport', specifies the lookup method 'by ICAO code', and lists specific output fields (name, IATA code, etc.), distinguishing it from siblings like get_aircraft or get_metar.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for retrieving airport details by ICAO code but lacks explicit guidance on when to use this tool versus alternatives or when not to use it. However, sibling tools have distinct resources, so context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_glossary_termA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It describes the lookup behavior and return fields, but does not explicitly state it is read-only or mention error handling, rate limits, or availability. The behavior is clear but lacks extra context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no waste. The first sentence states clear purpose with examples; the second lists return fields. Front-loaded and efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with one parameter and no output schema, the description is sufficiently complete. It explains the input, source, and output. Minor omission: could clarify that the term must be a slug, but the examples convey that.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema descriptions cover 100% of the parameter, but the description adds value by specifying the glossary source (Rotate Pilot), giving examples of slugs, and indicating the return fields. This goes beyond the schema description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it looks up a definition from the Rotate Pilot glossary by slug, with specific examples (squawk, angle-of-attack, v-speeds) and return fields (definition, category, related terms). It distinguishes itself from siblings like get_aircraft and get_metar.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for looking up aviation term definitions, but does not explicitly state when to use or not use this tool compared to alternatives like practice_questions or quiz_of_the_day.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• practice_questionsA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It explains the return of questions with options and correct answer, but does not mention whether the operation is idempotent, auth requirements, or rate limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with purpose and output, then filter and use case. No unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with 2 optional params and no output schema, the description adequately covers purpose, output, filtering, and intended use. No gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the description adds only marginal value via 'Optionally filter by subject'. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Get FAA-style aviation practice exam questions with options and the correct answer', which is a specific verb and resource. It is distinct from sibling tools like get_aircraft or get_airport.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description says 'Great for building study tools or quizzing a user', providing clear usage context. It does not explicitly list when not to use it, but the guidance is sufficient given the distinct sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• quiz_of_the_dayA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, but description fully discloses behavior: a read-only retrieval of a single question with answer and explanation. No side effects or state changes; transparency is adequate for a simple getter.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, front-loaded with key information, no extraneous content. Every word serves a purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no parameters, no output schema, and simple behavior, the description provides sufficient context: it explains the output format and content. No gaps for a tool of this complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters; description adds nothing beyond the empty schema. Baseline of 4 is appropriate as no parameter information is needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the tool retrieves the aviation question of the day, a single FAA-style multiple-choice question with answer and explanation. Distinguished from siblings like practice_questions which likely provide multiple questions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Implies usage for daily question retrieval but lacks explicit guidance on when to use this versus siblings like practice_questions, which might offer more questions or customization.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.