Server Quality Checklist

Tool Scores

• lookup_zipcodeA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It states inputs are allowed with hyphens and outputs multiple formats, but does not disclose behavioral traits like idempotency, error handling, or whether it modifies data. As a simple lookup, this is adequate but not fully transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is brief (two sentences) and front-loaded: first sentence states purpose, second provides key details. Every sentence adds value with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple input and lack of output schema, the description adequately lists return fields (prefecture, city, town in kanji/katakana/romaji, codes). However, it does not mention error cases (e.g., zipcode not found) or provide examples, which would improve completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% for the single parameter, but the description adds value by stating that hyphens are allowed, which is not explicitly in the schema's description. This clarifies input formatting beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: to look up a Japanese address for a 7-digit postal code. It lists the specific output fields (prefecture, city, town in multiple scripts and codes), distinguishing it from sibling tools like 'search_by_address' or 'list_cities_in_prefecture'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use (for Japanese address lookup by zipcode) and explicitly allows hyphens. However, it lacks explicit guidance on when not to use or alternatives, such as suggesting 'search_by_address' for broader address queries.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_cities_in_prefectureA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It mentions sharing an in-memory dataset with search_by_address, but does not explicitly state that the operation is read-only or disclose any limitations or side effects. For a simple list operation, this is adequate but not highly transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no redundant information. The first sentence states the core purpose, and the second adds important context about input formats and dataset sharing. Every part earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter, no output schema), the description covers what the tool returns, acceptable input formats, and its relationship to a sibling tool. No critical information is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and the parameter description already provides an example. The description adds value by specifying that the prefecture name can be in kanji, katakana, or romaji, which goes beyond the schema's example.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action (list) and resource (municipalities in a Japanese prefecture), and mentions the output includes Soumu city codes. It distinguishes the tool from siblings like search_by_address or lookup_zipcode.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for listing cities given a prefecture name, with flexibility in input format. It does not explicitly state when not to use or provide alternatives, but the context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_by_addressA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It discloses matching behavior (kanji, katakana, romaji, case-insensitive, whitespace ignored) and a critical performance trait (first call downloads ~25MB, subsequent calls instant). It does not mention auth or rate limits, but for a search tool this is adequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise (three sentences) and front-loaded: first sentence states purpose, second explains matching, third describes performance. Every sentence adds essential information with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers purpose, matching behavior, performance, and gives parameter examples. It does not explicitly detail the output format (e.g., whether returns full address or just zip), but the tool name implies postal codes. Given no output schema, this is a minor gap, but overall sufficient for an agent to use correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds value by explaining that 'query' is a free-text address keyword and providing concrete examples. The 'limit' parameter is already well-described in the schema. This extra context helps the agent generate appropriate queries.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches for postal codes by free-text address, and specifies matching against prefecture, city, and town in multiple scripts. It distinguishes itself from sibling tools (get_metadata, list_cities_in_prefecture, lookup_zipcode) by focusing on free-text address queries.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains that the tool is for free-text address queries and provides matching behavior (case-insensitive substring, whitespace ignored). It does not explicitly mention alternatives or when not to use, but the sibling tools imply different use cases. The performance note about dataset caching is helpful for usage planning.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_metadataA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, but the description lists all returned metadata fields, giving full transparency for a read-only operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences with no fluff, directly stating action and output.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no output schema, the description fully explains the return structure (version, timestamp, counts), making it complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has no parameters, so no additional parameter info is needed; the description suffices.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns dataset metadata (version, timestamp, counts) with specific fields, leaving no ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description includes 'Useful for confirming the dataset is current,' which implies when to use it, though it does not explicitly exclude alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.