Server Quality Checklist

Tool Scores

• search_datasetsC
  2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It does not disclose whether the operation is read-only, pagination behavior, or any side effects. The description is silent on important behavioral traits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very short but lacks the structure needed to convey essential information. It is terse but not concise in a helpful way, omitting critical details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 11 parameters, no output schema, and no annotations, the description is grossly insufficient. An agent cannot understand how to filter, paginate, or interpret results.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description must compensate. It only states 'list all datasets', but the schema includes 11 parameters for filtering, pagination, etc. No parameter meanings are explained.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a clear verb ('List') and resource ('datasets'), but lacks specificity about filtering capabilities. It does not distinguish itself from the sibling tool 'list_datasets', which likely has similar purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives like 'list_datasets'. No prerequisites, context, or exclusions mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_dataset_variable_dataC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must disclose behavioral traits. It only notes that 'Time is always included in the results,' but omits other critical aspects such as data ordering, pagination, limits, or whether the tool supports subsetting beyond time range. The lack of safety or mutation hints is a gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise at one sentence, but it lacks structure and omits essential details. While brevity is positive, the trade-off is missing critical information that would make it more useful.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 6 parameters, no output schema, and no annotations, the description is severely incomplete. It does not explain the return format, data types, error conditions, or how parameters interact. The agent cannot form a complete mental model of the tool's behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds no meaning to the 6 parameters, despite 0% schema coverage. It does not explain the purpose of erddap_url, dataset_id, variable_name, start_time, end_time, or exclude_nans. The agent must rely on parameter names alone, which may be insufficient.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves data for a variable from an ERDDAP dataset, using a specific verb ('Get') and resource ('data for a variable'). This distinguishes it from sibling tools that focus on metadata (list_datasets, list_dataset_variables) or search (search_datasets), making it obvious when to use this tool for actual data retrieval.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. For example, it does not explain that this tool is for fetching observations while list_dataset_variables is for discovering variable names. The agent must infer usage from the name and description alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_dataset_variablesC
  2.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It only states the action, lacking details on result format, pagination, authentication needs, or side effects. The name implies read-only, but this is not confirmed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single short sentence, which is concise but borderline under-specified. It could include additional context without becoming verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple operation and lack of output schema, the description should at least hint at what the tool returns (e.g., list of variable names). It does not, leaving the agent to guess the output format.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0% (no parameter descriptions). The description does not mention the two required parameters (erddap_url, dataset_id) or add any context beyond their titles. Fails to compensate for missing schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'List all variables for a dataset', using a specific verb and resource. It implicitly distinguishes from siblings like get_dataset_variable_data (which gets data for a specific variable) and list_datasets (which lists datasets).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives. The description does not mention prerequisites, typical use cases, or exclusions (e.g., not for querying data).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_datasetsC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, and the description does not disclose behavioral traits such as idempotency, rate limits, or authentication requirements. The tool likely performs a read operation but this is not explicitly stated.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with one sentence, front-loading the core purpose. However, it could be slightly more informative without becoming verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no output schema and one required parameter, the description does not explain the return format or how the URL is used. It leaves gaps for a simple tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The parameter 'erddap_url' is not mentioned in the description despite being required. With 0% schema description coverage, the description adds no meaning beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'list' and the resource 'all datasets in an ERDDAP server'. It distinguishes from sibling tools like 'search_datasets' by implying a full listing vs filtered search.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives like 'search_datasets' or 'list_dataset_variables'. No prerequisites or context provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_variable_standard_namesA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavioral traits. It does not mention any side effects, authentication needs, rate limits, or whether the list is complete or paginated. The read-only nature is implied but not explicit.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two paragraphs with the main purpose front-loaded. It is concise and to the point, with no unnecessary information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a listing tool with no output schema, the description explains the purpose of the output but does not describe its structure (e.g., list of strings, format of standard names). Some missing details, but overall adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The single parameter 'erddap_url' has no description coverage in the schema (0%), and the description does not explain its format or constraints. The parameter is simple, but without elaboration, the agent lacks guidance.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'List all variable parameter CF standard names', specifying the verb (list), resource (variable parameter standard names), and scope (datasets served by this ERDDAP server). It distinguishes from sibling tools like list_datasets and search_datasets.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains that this tool should be used to retrieve standard names before searching for datasets, providing clear context and a recommended workflow. It does not explicitly state when not to use it, but the guidance is strong.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.