Server Quality Checklist

Tool Scores

• describe_datasetA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses the return structure in detail, but does not explicitly state that the tool is read-only or discuss any side effects, auth requirements, or rate limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise (three sentences), well-structured, and front-loaded with the main action. Every sentence adds value, with no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the presence of an output schema, the description adequately covers the return format and use case. It lacks mention of edge cases or error scenarios, but for a simple descriptive tool, it is nearly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with examples and case-insensitivity. The description adds no additional parameter meaning beyond what the schema already provides, so baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Describe a dataset's filterable dimensions, returnable measures, units, and source.' It uses specific verbs and resource nouns, and distinguishes itself from siblings by positioning it as a prerequisite for 'get_data'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Use this before calling get_data on a new dataset', providing clear context for when to use. However, it does not explicitly mention when not to use it or compare with alternatives like 'search_datasets'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_datasetsA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully carries the burden. It discloses the return type and catalog contents, implying a read-only search operation. While it doesn't discuss authentication or rate limits, the provided context about fuzzy matching and ranked results adds value.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured, starting with the core purpose, followed by catalog context, usage examples, and return type. Every sentence contributes meaningfully.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is largely complete for a simple search tool, covering purpose, examples, return format, and catalog contents. An output schema exists, so detailed return specs are not needed. Missing discussion on error handling or empty results, but overall sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description's examples illustrate typical queries but add no substantial new semantics beyond the schema's parameter descriptions and examples for 'query' and 'limit'.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Fuzzy-search the curated AIHW dataset catalog' with a specific verb and resource. Examples and return type further clarify the tool's function, and it distinguishes itself from siblings like describe_dataset and get_data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for discovering datasets via free-text queries but lacks explicit guidance on when to use this tool versus alternatives like list_curated or top_n. No when-not-to-use or exclusion criteria are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_dataA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided; description covers return structure and filter behavior (OR across list values) but lacks info on rate limits, auth, or error handling. Decent but not comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Concise summary, illustrative examples, and return description—every element earns its place. Front-loaded purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers all key parameters (dataset_id, filters, measures, format, periods) and references describe_dataset for more detail. Output schema is mentioned, making it complete for a data query tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  100% schema coverage gives baseline 3; description adds value with examples, notes on plain-English aliases, and period formats, slightly exceeding schema alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it queries a curated AIHW dataset and returns observations, with specific verb and resource. Examples differentiate from siblings like search_datasets and latest.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Examples imply usage, but no explicit when-to-use or when-not-to-use compared to siblings. No alternatives mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• latestA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It discloses key behaviors: trimming to most-recent period, headline_slice defaults, user overrides. Does not cover auth or rate limits, but overall good.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Description is front-loaded with main purpose, then explains nuances. Examples are helpful. Could be slightly more concise, but well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given output schema exists and moderate complexity, description is thorough. Explains headline_slice, table type behavior, and provides examples. Complete for agent use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline 3. Description adds little beyond schema for parameters, only notes that filters are same shape as get_data and gives examples. No extra detail for measures.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description states the tool returns the most recent observation(s) per measure for a dataset. It distinguishes from siblings like get_data by focusing on latest data and explains behavior for different table types.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explains when to use for transposed vs wide tables, and how headline_slice applies. Provides examples. Lacks explicit when-not-to-use but context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_curatedA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, but the description discloses that the output is a sorted list of dataset IDs and characterizes the data (curated, with specific filter behavior). This is adequate for a simple read-only tool, though it lacks details on caching or live-fetching.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is only four sentences, each serving a purpose: stating the action, explaining the concept, pointing to related tool, and declaring the output format. No redundant or vague language.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (no parameters, clear output), the description covers its role and relationship to siblings. It could mention error cases or performance notes, but for a list command this level is sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the input schema coverage is 100%. Per guidelines, baseline 4 applies, and the description adds no parameter information as none is needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states the action (List) and the resource (every curated dataset ID), and distinguishes from siblings by explaining the meaning of 'curated'—datasets accepting plain-English filters—which sets it apart from describe_dataset and get_data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description hints at when to use this tool (to find curated datasets) and directs users to describe_dataset for documentation, but does not explicitly state when not to use it or compare directly to search_datasets.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• top_nA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It explains that the tool performs server-side ranking, returns at most n records sorted by measure, and that other fields match get_data. The read-only nature is implied. It could be more explicit about idempotency or side effects, but overall the behavioral traits are adequately disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is highly concise: a single sentence defining the tool, a paragraph explaining its value, three clear examples, and a returns summary. Every sentence serves a purpose, and the structure is front-loaded with the most critical information. No redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (5 parameters, no annotations, sibling tools present), the description is complete. The output schema is available (as per context), so detailed return value explanation is unnecessary. The description covers purpose, usage, examples, and behavioral traits sufficiently.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, so the baseline is 3. The description adds value by providing examples and usage guidance for each parameter, such as suggesting the describe tool for measure discovery and clarifying direction. This goes beyond the schema's basic descriptions, earning a 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the function: 'Return the N rows with the largest (or smallest) value of a measure.' It uses a specific verb (return) and resource (rows), and distinguishes itself from sibling tools like get_data by explaining that it avoids retrieving full tables. The purpose is unambiguous and well-defined.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly identifies the primary use case: 'show me the top 10 X by Y' and contrasts with get_data to highlight efficiency. It provides multiple examples illustrating typical usage. While it doesn't explicitly state when not to use, the context is clear. A slight enhancement could be mentioning to use describe_dataset for measure discovery, but the parameter description already hints at that.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.