Server Quality Checklist

Tool Scores

• fetch_fileA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the burden of behavioral disclosure. It explains that JSON files are returned as parsed objects while other files are returned as strings, and it gives path conventions with examples. This is a useful behavioral trait beyond the bare schema. It does not mention error handling, but for a simple fetch tool this is reasonably transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and well-structured, with a one-sentence summary followed by Args and Returns sections. Every sentence adds value: the purpose, the format behavior, the parameter explanation, and the return type. No fluff or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple one-parameter tool with no output schema, the description covers the essential aspects: what it does, how the path works, and what the return type is. It lacks error behavior or file size limits, but these are not critical for a straightforward fetch operation. The explanation of parsed JSON vs string return is particularly useful.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides only the parameter name and type with no description (0% coverage). The description fully compensates by explaining that 'path' is relative to the server root and provides concrete examples like 'json/solar_regions.json'. This adds significant meaning beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Fetch the content of a file from the SWPC data server', which is a specific verb+resource. It distinguishes itself from siblings like list_directory and describe_product by focusing on retrieving raw file content rather than metadata or listings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implicitly communicates when to use it: whenever you need the contents of a specific file. The context is clear from the verb 'Fetch' and the path parameter. It doesn't explicitly mention alternatives, but the purpose is distinct enough that no exclusions are needed.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_space_weather_scalesA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It discloses the return format and content in detail, including the G/S/R scales and solar flare classes. It does not explicitly state read-only status or lack of side effects, but the nature of a lookup tool makes this clear enough.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured: a clear opening sentence, a bulleted breakdown of scales, and a final returns line. Every sentence carries useful information, with no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  This is a zero-parameter reference lookup with an output schema. The description provides all necessary context, naming the returned dictionary keys and the substantive contents of the NOAA scales and solar flare classes, making it fully complete for an agent to decide and invoke.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the baseline is 4. The description need not explain parameter meanings, and it compensates by describing the returned data structure in detail.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Return the NOAA space weather scales and solar flare classification system.' It clearly distinguishes this from sibling file/product tools by describing a reference lookup, and explains the exact content returned.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Call this tool to understand how to interpret space weather severity,' giving clear context for when to use it. It does not mention alternatives or exclusions, but the sibling tools are unrelated, so no additional guidance is necessary.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_data_file_infoA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of disclosure. It explains the return structure (dict with description, fields, interpretation, related files) and the conditional behavior when a path is not found. This covers the main behavioral traits, though it omits details like error handling for malformed paths or access restrictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-organized: purpose, usage, argument, and return format. Every sentence adds value, and the structure is easy to scan.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has a single parameter, no annotations, and an output schema (though not detailed in the description). The description provides usage context, example paths, output structure, and fallback behavior, making it complete for selecting and invoking the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides only 'path' as a string with no description. The description compensates fully by explaining the path must be the same as used with fetch_file() and provides concrete examples ('json/planetary_k_index_1m.json'). This adds critical meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description begins with 'Return field definitions and interpretation guidance for a known data file,' which clearly states a specific action and resource. It also distinguishes itself from siblings by explicitly saying 'Use this tool before calling fetch_file()' and by mentioning it returns related files.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear guidance on when to use the tool ('Use this tool before calling fetch_file() to understand what the data means'), but it does not explicitly mention alternatives or exclusions. The fallback behavior of returning known paths hints at alternatives but doesn't name sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_directoryA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses the exact return structure ('dict with two keys'), the behavior for empty paths (list root), and the formatting of directory names (no trailing slash). It does not discuss error handling or permission issues, but for a simple listing operation, this is adequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with distinct sections for purpose, server context, arguments, and return value. Each part serves a purpose: the top-level directory examples help the agent form valid paths, and the return format avoids ambiguity. It is informative without being verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple listing tool with one optional parameter and an output schema present, the description is complete. It covers the root listing behavior, provides example paths, and explains the return value structure. No critical operational details are missing for the agent to invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0% because the input schema lacks descriptions. The description compensates fully by explaining the 'path' parameter with examples ('json/', 'json/goes/') and specifying that empty string lists the root. This adds significant meaning beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action ('List files and subdirectories at a path') on a specific resource (the SWPC data server). It distinguishes itself from siblings like fetch_file, which retrieves file contents, and list_products, which lists products rather than arbitrary directory contents.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context on when to use the tool: browsing the server's file tree, with examples of top-level directories and path formats. It doesn't explicitly name alternative tools or exclusions, but the usage context is unambiguous enough for an agent to distinguish this from file-fetching or product-listing tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_productsA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of disclosure. It does well by specifying the return structure (dict mapping categories to lists of product dicts with slug, name, description, data_paths) and the filter semantics. It does not mention edge cases like invalid category input or error behavior, but for a simple list operation this is solid.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the purpose, followed by compact, well-organized sections for available categories, arguments, and return shape. Every sentence earns its place and there is no padding or repetition of schema information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that this is a simple one-parameter list tool with an output schema and no annotations, this description provides everything an agent needs: clear purpose, filter behavior, return format, and category list. There are no significant gaps that would hinder selection or invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema only defines a defaulted string with no description, so the description fully compensates for 0% schema coverage. It explains that category is optional, case-insensitive, and that passing '' or omitting returns all categories. It also lists all valid category values, adding significant meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'List SWPC space weather products', and immediately adds the optional category filter. It states what each product entry includes (name, description, data paths), which clearly distinguishes it from sibling tools like list_directory or describe_product.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear usage context for the category argument: optional, case-insensitive, and empty/omitted returns all categories. It also enumerates valid category values. However, it does not explicitly name alternative sibling tools or state when not to use this tool, so it stops short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_known_pathsA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses that the tool returns a curated, grouped map and highlights useful files, which conveys read-only behavior and output structure. It stops short of explicitly stating 'no side effects' or auth requirements, but for a listing/orientation tool this is adequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured: a clear one-sentence purpose, a usage directive, and a brief Returns section. Every sentence adds value without redundancy or unnecessary length.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no parameters and an output schema is present, the description fully covers what an agent needs to select and invoke it correctly. It includes purpose, usage context, and output format, making it complete for this simple tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and the description's baseline for parameter semantics is therefore 4. There is nothing to add beyond the schema, and the description appropriately focuses on purpose and output rather than parameter details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb phrase 'Return a curated map of notable paths and files' and identifies the resource 'SWPC data server'. It clearly distinguishes the tool from siblings by stating it is a preliminary orientation tool, referencing list_directory and fetch_file as subsequent steps.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit guidance is provided: 'Use this tool first to orient yourself before calling list_directory or fetch_file.' This tells the agent exactly when to use the tool and names the alternatives, making tool selection unambiguous.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• describe_productA
  4.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of disclosing behavioral traits. It states the tool fetches a live product page and returns plain text, which implies a read-only network operation. It does not detail error handling or auth, but the core behavior is transparent. It slightly falls short of a 5 by not explicitly mentioning it does not modify anything, but 'fetch' strongly implies that.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a clear opening sentence, a usage note, and explicit Args/Returns sections. Every sentence contributes value: purpose, prerequisite, behavior, and return format. It is appropriately sized for the tool's moderate complexity without wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is complete given the tool's simplicity: one parameter, no nested objects, and an output schema (which covers return values). It explains what the tool does, how to use it, what input to provide, and what output to expect. It also provides contextual rationale ('useful for detailed interpretation guidance'). No significant gaps remain.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description fully compensates. The only parameter, product_slug, is thoroughly explained: 'The product URL slug, e.g. "planetary-k-index"...' with concrete examples. This adds meaning well beyond the bare schema definition.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Fetch the full description of a SWPC product from the NOAA website.' This specifies the verb (fetch), resource (SWPC product description), and outcome (full description). It distinguishes from siblings like list_products (which lists products) and get_data_file_info (which fetches data), making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly advises to 'Use list_products() to find product slugs,' naming a sibling tool for a prerequisite step. It also clarifies when this tool is useful: 'for detailed interpretation guidance, methodology, and caveats.' This gives clear context and a direct alternative, meeting the 'when to use' requirement.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.