Server Quality Checklist

Tool Scores

• getDatasetInfoAndSchemaC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. The description only states what the tool does ('Get dataset info and schema') without explaining behavioral traits such as whether it's a read-only operation, what permissions are needed, how it handles errors, or what format the output takes. This leaves significant gaps in understanding the tool's behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise ('Get dataset info and schema') and front-loaded with the core purpose. It uses minimal words without unnecessary elaboration, though it could be more structured by including key details. Every sentence (here, just one) earns its place by stating the tool's function.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (a read operation with one parameter) and the absence of annotations and output schema, the description is incomplete. It doesn't explain what 'info' includes (e.g., metadata, statistics) or the schema format (e.g., JSON, table structure), leaving the agent with insufficient context to use the tool effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, with the single parameter 'dataset' documented as 'The dataset to get info and schema for'. The description adds no additional meaning beyond this, as it doesn't elaborate on what constitutes a valid dataset name or provide examples. With high schema coverage, the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the tool's purpose ('Get dataset info and schema') which is clear but vague. It specifies the verb 'Get' and resource 'dataset info and schema', but doesn't distinguish it from sibling tools like 'listDatasets' or 'queryApl'. The purpose is understandable but lacks specificity about what 'info' includes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. There's no mention of when to use 'getDatasetInfoAndSchema' instead of 'listDatasets' or 'queryApl', nor any context about prerequisites or exclusions. The user must infer usage from the tool name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• listDatasetsB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It states the action ('List all available') but doesn't cover critical traits like pagination, rate limits, authentication needs, or what 'available' means (e.g., permissions, status). This leaves significant gaps for a tool that likely interacts with datasets.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with zero waste—'List all available Axiom datasets' directly conveys the core purpose without extraneous details. Every word earns its place, making it highly efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (0 parameters, no output schema), the description is minimally adequate but lacks depth. Without annotations or output schema, it should ideally mention return format (e.g., list of dataset names/IDs) or behavioral constraints. It's complete enough for basic understanding but misses contextual nuances.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0 parameters with 100% coverage, so no parameter documentation is needed. The description appropriately doesn't discuss parameters, earning a baseline high score. It could be a 5 if it explicitly noted the lack of parameters, but the current brevity is sufficient.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('List') and resource ('available Axiom datasets'), making the purpose immediately understandable. It doesn't distinguish from sibling tools like getDatasetInfoAndSchema or queryApl, which prevents a perfect score, but the core functionality is well-defined.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like getDatasetInfoAndSchema (for detailed info) or queryApl (for querying data). It lacks explicit context, prerequisites, or exclusions, leaving the agent to infer usage from tool names alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• queryAplA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It thoroughly describes key traits: the query must be a valid APL string, there's a maximum row limit of 65000 rows, performance considerations (e.g., reducing data scanned by restricting time ranges), and cost implications (e.g., 'It's expensive to project all fields'). This goes well beyond the basic input schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is excessively long and poorly structured for a tool definition. It includes extensive examples and patterns that belong in documentation rather than a concise tool description. The front-loaded instructions are useful, but the bulk of the text (examples) adds unnecessary verbosity without earning its place in a tool description meant for AI agents.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of querying datasets with no annotations or output schema, the description is largely complete. It covers purpose, usage, behavioral traits, and parameter semantics thoroughly. However, the lack of output schema means the description doesn't explain return values or result formats, and the over-inclusion of examples reduces focus on core completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% coverage for the single parameter 'query', describing it as 'The APL query to run.' The description adds significant semantic context by explaining what constitutes a valid APL query (e.g., examples of syntax, operators, and patterns) and constraints (e.g., must be valid APL). However, it doesn't explicitly link this to the parameter name or format beyond the examples.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Query Axiom datasets using Axiom Processing Language (APL).' It specifies the verb ('query'), resource ('Axiom datasets'), and technology ('APL'), distinguishing it from sibling tools like getDatasetInfoAndSchema and listDatasets, which likely retrieve metadata rather than execute queries.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when and how to use this tool, including prerequisites (e.g., 'ALWAYS get the schema of the dataset before running queries'), best practices (e.g., 'Prefer aggregations over non aggregating queries'), and alternatives (e.g., 'use search first to find in which fields it appears'). It also specifies exclusions like avoiding schema guessing and limiting time ranges.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.