Server Quality Checklist

Tool Scores

• analyze_datasetC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description does not disclose what 'statistical analysis' entails—whether it modifies the dataset, requires permissions, or what side effects occur. The behavior is opaque.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very concise (6 words) but at the expense of informative content. It is front-loaded but insufficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the output schema exists, the description should briefly indicate what kind of statistics are returned. It does not, leaving the agent uncertain about the tool's output.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and the description adds no additional meaning to the 'dataset_id' parameter beyond the schema's description. Baseline score applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool performs statistical analysis on a dataset, distinguishing it from sibling tools like list_datasets or query_dataset. However, it lacks specificity on the type of analysis (e.g., descriptive, inferential).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is given on when to use this tool versus alternatives like query_dataset for raw data retrieval. The description does not specify contexts or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_datasetsB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are present, so the description must convey all behavioral traits. It only states the action without mentioning return type, pagination, or any side effects. The agent gains minimal insight into behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no unnecessary words. It efficiently communicates the core purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the existence of an output schema, the description doesn't need to detail return values. However, it omits any mention of scope, filtering behavior, or how results are ordered, making it minimally complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The single parameter 'query' is already well-described in the input schema with examples. The description adds no extra meaning beyond what the schema provides, meeting the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it searches datasets in a specific catalog. However, it does not differentiate itself from the sibling tool 'list_datasets', which could be confused as a similar operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use search vs. other tools like 'list_datasets' or 'query_dataset'. The description lacks context for appropriate usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• query_datasetB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, and the description lacks details on behavioral traits (e.g., read-only nature, pagination, error handling). The description carries full burden but only mentions optional filtering.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise (one sentence) and front-loaded, earning its place. However, it could be slightly more informative without bloat.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the existence of an output schema and 4 parameters, the description is minimal but covers the core action. Lacks context about return format or limitations, but output schema compensates partly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema covers 100% of parameters with descriptions. The tool description adds 'with optional filtering' which summarizes filter parameters but does not add new meaning beyond schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Query dataset contents with optional filtering' clearly specifies the action (query) and resource (dataset contents) and distinguishes from siblings like analyze_dataset (analysis) and list_datasets (listing).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus siblings such as search_datasets or analyze_dataset. The description does not provide context for appropriate usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_dataset_schemaB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are present, so the description carries full burden for behavioral disclosure. It only states the retrieval action without indicating side effects, read-only nature, permissions needed, or any constraints. Minimal disclosure for a safe retrieval tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence with no extraneous information. It is front-loaded and every word contributes to understanding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has an output schema (not shown but flagged), the description does not need to detail return values. However, it lacks any mention of what 'schema information' entails (e.g., columns, types) and provides no context for integration with sibling tools. Adequate but not rich.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The single parameter 'dataset_id' is fully described in the input schema (100% coverage). The description adds no additional meaning beyond the schema's description, so baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves schema information for a specific dataset using a specific verb ('Get') and resource ('schema information for a specific dataset'). It distinguishes itself from sibling tools like analyze_dataset and query_dataset.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives (e.g., analyze_dataset, list_datasets). No prerequisites or exclusions are mentioned, leaving the agent to infer usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_datasetsA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description must carry full burden. It discloses that the tool returns a list of datasets with IDs, names, and descriptions, but does not elaborate on any other behavioral aspects (e.g., rate limits, safety). For a simple listing, this is adequate but minimal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences. The first sentence states the core purpose, and the second adds detail and usage context. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with no parameters and an output schema, the description explains what the output contains (IDs, names, descriptions) and the tool's role (discovery). This is fully complete given the context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has no parameters and schema coverage is 100%, so the baseline is 4. Description does not need to add parameter info.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'list' and resource 'datasets from Utsunomiya city', and distinguishes from sibling tools like query_dataset by indicating this is for discovery before querying.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says to use this before querying to discover available data, providing clear context. It lacks explicit exclusion of when not to use, but the guidance is sufficient.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.