Server Quality Checklist

Tool Scores

• get_dataA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must bear the full transparency burden. It states it returns observations and lists return fields but does not disclose read-only behavior, rate limits, or error conditions. While informative on output, it lacks explicit behavioral traits beyond what is obvious.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with a clear purpose statement, but the inclusion of four example blocks and a returns list makes it longer than necessary. While examples are helpful, the description could be more concise to improve readability.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (6 parameters, output schema present), the description provides examples and lists return fields, which is nearly complete. It does not mention pagination or limits, but the return structure is adequately described. Overall, it is well-rounded for a data retrieval tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description adds examples that illustrate parameter usage, but the schema already contains detailed descriptions and examples. The additional value is marginal; the description does not significantly clarify parameter semantics beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool queries a curated APRA dataset and returns observations, using a specific verb ('Query') and resource ('APRA dataset'). This distinguishes it from siblings like search_datasets (discovery) and describe_dataset (schema).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides multiple examples showing common use cases but does not explicitly state when to use this tool versus alternatives. Sibling names imply differentiation, but no direct 'when-not' or alternative references are given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• describe_datasetA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must bear the burden. It states the tool returns a DatasetDetail with specific fields, implying a read-only operation. However, it does not mention authentication, rate limits, or safety guarantees, which would enhance transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with paragraphs and bullet points, effectively front-loading the purpose. It is concise with no redundant information, though slightly verbose in listing return fields.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the single parameter and presence of an output schema, the description sufficiently explains the tool's purpose, usage, and return structure. It covers the key aspects for an agent to correctly invoke it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, with dataset_id having a description and examples. The description adds context (case-insensitive, curated) but does not significantly augment the schema beyond that. Baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool describes a dataset's filterable dimensions, measures, units, source, and framework break info. It uses specific verbs and resources, distinguishing it from siblings like get_data and search_datasets.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly advises using this tool before get_data on a new dataset, detailing the information it provides. While it does not list explicit exclusions or alternatives, the guidance is clear and contextually sufficient.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_curatedA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must disclose behaviors. It says the tool returns a sorted list, which implies a read operation, but does not mention potential performance impacts, auth needs, or pagination (if any). For a simple list tool, this is minimally adequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences with no fluff: first states purpose and scope, second states return format. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given zero parameters and an output schema, the description covers the essential information: it lists all curated dataset IDs, sorted. No gaps remain for an agent to understand its function.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are no parameters, so schema coverage is 100%. The description adds no extra meaning, but baseline for 0 parameters is 4. No improvements needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists every curated dataset ID, using the specific verb 'List' and specifying the resource. It distinguishes itself from siblings like describe_dataset (which describes one) and search_datasets (which filters), even without explicit comparison.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is given on when to use this tool versus alternatives such as search_datasets or top_n. The description simply states what it does, without any context for selection or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_datasetsA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description explains that the search is fuzzy, case-insensitive, and returns ranked results. It does not explicitly state read-only or safe behavior, but the examples and return type imply no destructive side effects. Could be more transparent about permissions or performance.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a clear purpose, list of included datasets, examples, and return type. It is slightly verbose but each section serves a purpose. Front-loading the main verb and resource is effective.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the presence of an output schema and the description's explanation of return fields (DatasetSummary), the description is fairly complete. It covers the tool's scope, usage, and examples, though it could benefit from mentioning if there are any limitations or prerequisites.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with clear descriptions for both 'query' and 'limit' parameters. The tool description adds examples and context about the catalog but does not significantly enhance understanding beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it's a fuzzy-search tool for a curated APRA dataset catalog, lists example datasets, and provides usage examples. It distinguishes from siblings like describe_dataset and list_curated by focusing on discovery via free-text search.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use (to find datasets) but does not explicitly contrast with sibling tools or state when not to use it. No direct guidance on alternatives, leaving the agent to infer context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• latestA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided; the description fully explains the trimming logic, row cap for wide-layout datasets, and truncated response indicator, exceeding basic expectations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Every sentence adds value: purpose, trimming logic, row cap note, and example. No redundancy, front-loaded with core action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers purpose, use case, behavior, and example adequately. With an output schema existing, return values need no further explanation. Minor improvement could include error handling, but not essential.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds context for limit (capping, truncation), filters (same shape as get_data), and measures (referencing get_data), raising it to 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states 'Return the most recent observation per measure for a dataset.' with examples like 'Latest CBA capital ratios,' clearly distinguishing it from siblings such as get_data (raw data) and top_n (top values).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description advises use for 'current value' queries without specifying periods, but does not explicitly exclude alternatives or list when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• top_nA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully discloses behavior: server-side ranking, limited rows, sorted by measure in requested direction, and return shape matching get_data. No contradictions or omissions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Sparse yet complete: one sentence for purpose, concise explanation, three clear examples, and one-line return note. Every sentence contributes meaning with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers workflow, parameters, examples, return behavior, and even references related tools (describe). Given five parameters and output schema, this is fully self-contained guidance.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, but description adds significant value with practical examples, context (e.g., pinning a period), and references to describe tool. The inline examples demonstrate usage tying parameters together.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it returns N rows with largest/smallest value of a measure, using specific verbs and resource. It distinguishes itself from siblings like get_data by focusing on ranking and limiting results server-side.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description calls it the 'single most common agent workflow' and provides examples, implying when to use. However, it lacks explicit guidance on when not to use or alternatives, which would elevate it to a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.