Server Quality Checklist

Tool Scores

• validate_asm_schemaA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It states the tool returns a JSON string with validation result but does not disclose whether the operation is read-only or has side effects, nor any error behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a compact docstring with clear Args and Returns sections. Every line serves a purpose and there is no redundant text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple validation tool with an output schema, the description covers purpose, parameters, and return type. However, it lacks usage context relative to sibling tools, though that is more of a usage guideline issue.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds 'File path to' for both parameters, clarifying that they are file paths beyond the schema's titles. This is minimal but meaningful for a simple two-string schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Validate an ASM JSON document against an Allotrope JSON Schema,' using a specific verb and resource. It distinguishes from siblings like validate_field_map which targets a different resource.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool vs alternatives like validate_field_map or fetch_asm_document. Usage is implied by the description but no alternatives or exclusions are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• describe_asmA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses the success return value (JSON string) and failure behavior (error object with list of valid names), which is helpful and goes beyond the schema. It does not mention permissions or side effects, but for a read-only describe operation this is acceptable, so 4.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured, with a clear purpose sentence followed by Args and Returns sections. Every part adds value, and it fits in a small space, so 5.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is complete for the tool's complexity: it states the purpose, explains the parameter, and explicitly specifies success and failure return formats. The presence of an output schema is not strictly necessary because the returns are described, and the tool is simple, so 5.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides only a type and requirement for model_name with no description. The description's Args section adds semantic meaning by labeling it as 'the key identifying the ASM model' and providing an example, which compensates for the 0% schema coverage, so 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Return the metadata for a specific ASM model by name,' using a specific verb and resource. However, it does not explicitly differentiate from sibling tools like fetch_asm_document or list_asms, so it earns a 4 rather than 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no guidance on when to use this tool versus alternatives. The description only explains what it does, not when to choose it over siblings like fetch_asm_document or list_asms, so it scores 2.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_asmsA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full disclosure burden. It reveals that data comes from a local bundled model_reference.json, describes the return format as a JSON string mapping IDs to descriptions, and notes the error-key fallback. This provides a solid behavioral profile for a simple read-only list operation, though it does not explicitly state non-mutating behavior (implied by 'list').

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded: a one-sentence summary, a single clarifying sentence about the data source, and a concise Returns block. Every sentence adds value, with no redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a parameterless listing tool, the description is complete: it states what is listed, where the data comes from, and what the output looks like including error behavior. The presence of an output schema is not an issue because the description already covers the return value, and no additional context is needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so schema coverage is effectively 100%. The description has no parameter details to add, and the baseline of 4 applies because there is nothing to compensate for. No ambiguity exists.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a clear, specific verb+resource: 'List all available Allotrope Simple Models (ASMs).' It further clarifies scope by mentioning it retrieves identifiers and descriptions from a local file, and the sibling tool names (validate, fetch, describe) contrast with this listing function.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for enumerating available ASMs and their metadata, but it does not explicitly state when to use this over sibling tools like fetch_asm_document or describe_asm, nor does it mention any exclusions or alternatives. The usage context is clear from the purpose, but explicit guidance is missing.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• validate_field_mapA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure, and it excels. It details the exact comparison logic (string equality primary, numeric float fallback), the file format (JSON), the return structure (match counts, mismatches, summary message), and error handling ('error' key on failure). This gives an agent a comprehensive understanding of the tool's behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a purpose statement, a detailed behavior section, and an Args/Returns breakdown. It is concise, front-loaded with the core purpose, and every sentence contributes valuable information without redundancy. This is a model of clear, efficient documentation.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity with one parameter and an output schema, the description provides all essential context: what the tool does, how it operates, what it returns, and error behavior. The presence of an output schema means return values are additionally specified, but the description already covers them. No significant gaps remain.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must explain the parameter, and it does: 'field_map_path: File path to the field mapping JSON file.' This adds meaning beyond the bare schema definition by specifying what the path refers to and that it is a JSON file. While comprehensive for one parameter, it leaves out details like file size limits or path format, so a 4 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Validate a field mapping file produced by a custom converter script,' which clearly specifies the verb (validate) and resource (field mapping file). This distinguishes it from siblings like validate_asm_schema, which validates schemas rather than field maps. The purpose is explicit and unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context that this tool is intended for files produced by a custom converter script, indicating when it is appropriate to use. However, it does not explicitly mention alternatives or when not to use it, such as if the user needs to validate an ASM schema (validate_asm_schema). This is a minor omission, hence a 4 rather than 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• fetch_asm_documentA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description fully discloses behavioral traits: it downloads, saves to a path mirroring the URI, returns a cached result immediately, and does NOT resolve $ref references. This goes beyond basic purpose and is highly transparent about side effects and limitations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a one-line summary, followed by details on behavior and parameters. Every sentence adds valuable information without redundancy, and it is front-loaded with the core purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (local file write, caching, URI handling) and lack of annotations, the description covers all essential aspects: what it does, how it saves, caching behavior, $ref limitation, parameter formats, and return structure. It is complete for an agent to select and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate. It does so by explaining asm_document_uri as a fully-qualified URI starting with http://purl.allotrope.org, and output_dir as the base directory defaulting to the current working directory. This adds meaning beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool fetches and downloads a raw ASM JSON document from purl.allotrope.org, saving it to the local filesystem. The verb 'Fetch' and resource 'document' are specific, and it distinguishes itself from sibling tools like validate_asm_schema or list_asms by focusing on raw retrieval.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context about when to use the tool (to fetch raw documents) and important caveats (caching, $ref not resolved), but it does not explicitly mention alternatives or when-not-to-use scenarios. The context is strong enough to guide usage without exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.