Server Quality Checklist

Tool Scores

• get_catalogue_dataC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It mentions fetching data but lacks details on behavioral traits such as authentication requirements, rate limits, error handling, or response format. This is a significant gap for a tool that interacts with an external API, making it inadequate for informed use.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's purpose without unnecessary words. It is front-loaded and appropriately sized, making it easy for an agent to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of fetching data from an external API, no annotations, and no output schema, the description is incomplete. It doesn't cover key aspects like authentication, rate limits, error responses, or data format, leaving the agent with insufficient context for reliable use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters ('id' and 'limit') with clear descriptions. The description adds no additional meaning beyond what the schema provides, such as examples or constraints, but the baseline is 3 since the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('fetch') and resource ('actual data from the data.gov.my API for a specific catalogue'), distinguishing it from siblings like 'get_catalogue_metadata' (metadata vs. data) and 'list_catalogue_ids' (list IDs vs. fetch data). However, it doesn't explicitly differentiate from 'search_catalogues', which might also fetch data but with search capabilities.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like 'search_catalogues' or 'get_catalogue_metadata'. The description implies usage for fetching data for a specific catalogue, but it doesn't specify prerequisites, exclusions, or contextual recommendations, leaving the agent to infer usage scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_cataloguesC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden but only states the basic search function. It doesn't disclose behavioral traits such as whether it's read-only, potential rate limits, authentication needs, or what the output format might be (e.g., list of results, pagination).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero waste. It is appropriately sized and front-loaded, directly stating the tool's purpose without unnecessary elaboration.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations and output schema, the description is incomplete. It covers the basic purpose but fails to address key contextual aspects like return values, error handling, or behavioral constraints, which are crucial for a search tool with no structured support.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the input schema fully documents the 'keyword' parameter. The description adds minimal value beyond the schema by reiterating the search scope ('in catalogue names/descriptions'), but no additional syntax or format details are provided.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Search for catalogues') and the resource ('catalogues'), with specific search scope ('by keyword in their names or descriptions'). It distinguishes from siblings like 'get_catalogue_data' or 'list_catalogue_ids' by emphasizing search functionality, though it doesn't explicitly contrast them.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like 'get_catalogue_data' or 'list_catalogue_ids'. The description implies usage for keyword-based searches but lacks explicit context, prerequisites, or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_catalogue_metadataB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the action is to 'fetch metadata,' implying a read-only operation, but doesn't clarify aspects like authentication requirements, rate limits, error handling, or what metadata fields are returned. This leaves significant gaps for a tool interacting with GitHub.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core action and resource. It wastes no words and is appropriately sized for a simple tool with one parameter, earning a high score for conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (one parameter, no output schema, no annotations), the description is minimally adequate. It covers the basic purpose but lacks details on usage guidelines, behavioral traits, and output expectations, which are needed for full contextual understanding despite the simple schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, with the 'id' parameter well-documented in the schema itself. The description adds no additional meaning beyond implying the ID is for a data catalogue, which is already clear from the schema. This meets the baseline score of 3 when schema coverage is high.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('fetch metadata') and resource ('specific data catalogue by ID from GitHub'), making the purpose immediately understandable. However, it doesn't differentiate this tool from its siblings like 'get_catalogue_data' or 'search_catalogues', which likely have overlapping domains but different functions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It mentions fetching by ID but doesn't explain when to choose this over 'list_catalogue_ids' (for listing IDs) or 'search_catalogues' (for broader searches), leaving the agent to infer usage from tool names alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_catalogue_idsB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It states 'Fetch list' which implies a read operation, but doesn't disclose behavioral traits like whether it requires authentication, has rate limits, returns paginated results, or what format the list comes in. The GitHub repository mention adds minimal context but insufficient for a mutation-free tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the key information ('Fetch list of available data catalogue IDs') with no wasted words. Every part of the sentence contributes to understanding the tool's purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 0 parameters, no annotations, and no output schema, the description is minimally adequate but lacks completeness. It specifies the source (GitHub repository) and what's fetched (IDs), but doesn't cover behavioral aspects like response format or error handling, leaving gaps for a read operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters with 100% schema description coverage, so the schema fully documents the lack of inputs. The description doesn't need to add parameter semantics, and it appropriately doesn't mention any parameters. A baseline of 4 is justified since no parameters exist to explain.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Fetch list') and resource ('available data catalogue IDs from GitHub repository'), providing a specific verb+resource combination. However, it doesn't explicitly differentiate from sibling tools like 'search_catalogues' or 'get_catalogue_metadata', which might also retrieve catalogue-related information.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'search_catalogues' or 'get_catalogue_metadata'. It mentions fetching from a GitHub repository, which gives some context, but lacks explicit when/when-not instructions or named alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.