Server Quality Checklist

Tool Scores

• get_dataA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It describes the return type (DataResponse with fields) and mentions that it queries data without side effects. However, it lacks explicit statements about read-only nature, authentication, rate limits, or error behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a clear opening sentence and bullet-pointed examples. While longer due to examples, each sentence adds value and the structure aids readability.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (5 parameters, output schema exists), the description covers the return type, common filter patterns, and period formatting. It could mention pagination or error handling but is otherwise adequate for agent use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds significant value through extensive examples showing how parameters combine in real queries (e.g., filters with start/end periods). This clarifies usage beyond schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Query') and resource ('AEMO NEM dataset'), with multiple examples showing typical use. While it doesn't explicitly distinguish siblings, the examples implicitly separate it from search, describe, and list tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides examples that illustrate common use cases but does not explicitly state when to use this tool versus alternatives like search_datasets or describe_dataset. Usage is implied rather than directly guided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• describe_datasetA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description must be self-contained. It lists the return fields (id, name, description, filters, units, source URL, example invocation strings) and includes an example with return structure. No behavioral contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise: one sentence for purpose, followed by an example and return format. No redundant information; each sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter, output schema present), the description covers all essential information: what it does, how to use it, and what it returns. No gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides comprehensive details (100% coverage) for the single parameter, including examples and case-insensitivity note. The description does not add extra parameter semantics beyond mentioning the dataset types, which is not necessary.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description starts with 'Describe one NEM dataset — schema, filters, cadence, source URL' which clearly states the action and resource. It differentiates from sibling tools like search_datasets and list_curated by focusing on a single dataset's metadata.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description advises using search or list-curated endpoints to discover dataset IDs before calling this tool. This gives clear guidance on the proper workflow, though it does not explicitly state when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• latestA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description provides good behavioral details: interval cadence per feed type, delay characteristics, and the stale flag. Lacks explicit mention that it is a read-only operation, but the context is clear.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-organized with sections for different feed types, a concise list of examples, and no unnecessary words. Every sentence contributes to understanding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that an output schema exists, the description provides sufficient context about the return type (DataResponse with stale flag) and covers all relevant feed types. No gaps identified.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has 100% coverage with descriptions and examples, but the description adds value by explaining the meaning in context, such as how filters narrow results and which dataset IDs apply.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description precisely states the tool's purpose: 'Return the most recent interval(s) for a NEM dataset.' It differentiates from siblings like get_data (historical) and search_datasets by focusing on the latest single observation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly tells when to use the tool (to get the most recent interval) and provides examples. However, it does not explicitly mention when not to use it or contrast with siblings like get_data for historical queries.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_curatedA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It reveals that the tool returns a sorted list of 7 entries and provides examples. It does not discuss error handling or immutability, but for a simple read-only list, this is adequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, front-loading the main purpose, and provides a clear example and summary. Every sentence adds value without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simplicity of the tool, the description is complete. It explains what the tool does, what it returns, and its typical use. The output schema is present, but the description already covers the return structure.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has no parameters, so the description adds value by explaining the output and purpose. Baseline for 0 parameters is 4, and the description appropriately compensates for the lack of parameter details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states that this tool lists the 7 curated AEMO NEM dataset IDs, which covers ~95% of typical queries. It distinguishes itself from sibling tools like 'get_data' and 'search_datasets' by being a no-parameter listing operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains that the tool returns the most common datasets, implying it should be used first for typical analytical queries. However, it does not explicitly state when not to use it or compare with alternatives like 'search_datasets'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_datasetsA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description explains that the tool returns a list of DatasetSummary objects ranked by relevance and that datasets are curated. It also provides example output structure. However, it does not address behavior like rate limiting or error handling, but for a search tool, the provided detail is sufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise yet comprehensive: it starts with a clear purpose, provides usage guidance, includes illustrative examples, and states the return format. Every sentence adds value, and the structure is logical and easy to scan.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (2 parameters, output schema exists), the description covers all essential aspects: purpose, when to use, return format, and examples. The output schema handles detailed return field definitions, so the description is complete. No omissions for typical usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already covers both parameters with descriptions and examples (100% coverage). The description adds value by explaining that the search is fuzzy, results are ranked by relevance, and the curated nature of the datasets. This context enriches the parameter meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states that the tool performs fuzzy-search on the 7 curated AEMO NEM datasets, and specifies it is for use when exact dataset_id is unknown. This distinguishes it from siblings like describe_dataset (which requires a known ID) and list_curated (which lists all curated datasets). The purpose is specific and actionable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly advises using this tool when the exact dataset_id is unknown, and mentions that curated datasets cover ~95% of typical NEM queries. This provides clear context for when to use this tool versus alternatives. Examples further illustrate usage scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.