Server Quality Checklist

Tool Scores

• get_healthC
  2.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description only mentions 'read-only', but with no annotations, more behavioral context is needed. It does not disclose authentication requirements, rate limits, or potential errors.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is short and includes an example, but it omits valuable details about parameters and usage. It is not verbose but could be more informative in the same space.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple interface (1 optional parameter, no output schema), the description is incomplete. It lacks parameter explanation and return value details, leaving the agent to guess.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description fails to mention the product_code parameter or its default value. With 0% schema description coverage, the description should compensate but does not.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it retrieves the operating status of the exchange and provides example status values, making the tool's purpose unmistakable and distinct from sibling tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives like get_board or get_ticker. No when-not-to-use or prerequisite information is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_board_stateB
  3.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description carries the full burden. It declares read-only behavior and lists example states, offering some transparency. However, it does not disclose rate limits, auth requirements, or behavior on errors.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very concise with three short sentences. The first sentence front-loads the core purpose. Every sentence adds value: purpose, state examples, and usage context.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with one parameter and no output schema, the description covers the basics: what it returns and a use case. However, it lacks details on return format or any caveats, limiting completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% description coverage and no enums. The tool description does not mention the 'product_code' parameter at all, failing to add any meaning beyond the schema's default value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves board state (health, state, special signs) and gives examples. It distinguishes from sibling tools like 'get_board' (full board data) and 'get_health' by focusing on state, though no explicit comparison is made.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides a usage context: 'Can be used to determine if trading is possible.' However, it does not explain when to use this tool over siblings or list any exclusions or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_executionsA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden. It discloses the read-only nature, output fields, and count range/default, but does not mention error handling, authentication needs, or rate limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with two sentences, front-loading the purpose and then detailing fields and count. No unnecessary words, though it could be slightly more structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description covers the return fields and count parameter. However, it omits 'product_code' and does not address ordering or pagination, making it somewhat incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%. The description explains the 'count' parameter (range 1-100, default 20) but does not mention 'product_code' at all, leaving it undocumented.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves recent trade execution history, specifies it is read-only, and lists the returned fields (price, size, side, exec_date). It distinguishes itself from sibling tools like get_board or get_ticker which serve different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description indicates it is read-only but provides no explicit guidance on when to use this tool versus alternatives, nor any exclusions or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_tickerA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description explicitly marks the tool as read-only and lists the main return fields, which is helpful given no annotations. However, it does not disclose potential rate limits, error conditions, or other behavioral details that could affect invocation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with two sentences and a bulleted list. It front-loads the purpose and read-only nature, then efficiently adds parameter examples and return value details with no redundant text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple ticker tool, the description adequately covers purpose, parameters, and return values despite lacking an output schema. It does not address error handling or limitations, but the core information is present and actionable.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The sole parameter product_code has a default but no schema description. The tool description compensates by providing specific examples (FX_BTC_JPY, BTC_JPY, etc.), adding meaningful context beyond the schema definition.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it retrieves the latest ticker from bitFlyer, specifies it is read-only, and lists the main return values. However, it does not explicitly differentiate from sibling tools like get_board or get_executions, though the specific resource ('ticker') implies uniqueness.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides product code examples but offers no guidance on when to use this tool versus alternatives (e.g., get_board for order book data). There is no mention of context or preconditions for use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_boardA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses that the tool is read-only and compresses the full order book to a specified depth (default 10) because the full board has hundreds to thousands of items. This goes beyond the bare minimum, though it omits details like authentication or rate limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise, with every sentence providing essential information. It front-loads the core purpose and return values, with no unnecessary words or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple read-only tool with only two parameters and no output schema, the description covers the key return fields (mid_price, spread, best_bid, best_ask, bids/asks) and explains the compression behavior. It is sufficient for an agent to invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has no parameter descriptions (0% coverage), but the tool description provides example product codes for product_code and explains that depth controls the number of levels for compression. This adds meaningful context that the schema alone does not provide.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves the order book from bitFlyer, summarized to top depth levels, and marks it as read-only. It identifies the specific resource and action, though it does not explicitly differentiate from sibling tools beyond the nature of the data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for getting a summarized order book, but does not provide explicit guidance on when to use this tool versus alternatives like get_ticker or get_board_state. There are no when-not-to-use instructions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_marketsA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description explicitly states the tool is read-only, which is a key behavioral trait. As there are no annotations, this adds value. For a simple list retrieval, it adequately discloses behavior, though it omits potential details like return format.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single, front-loaded sentence with no wasted words. Every part is necessary and informative.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no parameters, no annotations, and no output schema, the description covers the basic purpose. However, it could mention the expected return type or structure, making it slightly incomplete for an agent that might need to parse the result.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are no parameters, so the baseline is 4. The description does not need to add any parameter meaning beyond what the schema provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it retrieves a list of all tradable products on bitFlyer, specifying the types (spot, FX, futures) and that it's read-only. It distinguishes from sibling tools like get_board or get_ticker by focusing on the market list.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context: use when you need the full list of available markets. While no explicit when-not or alternatives are given, the read-only nature and clear purpose make it easy to decide.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.