Server Quality Checklist

Tool Scores

• dataprobe_query_sqlC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It does state 'read' indicating a non-mutating operation, but it does not disclose return format, pagination behavior, side effects, or authorization requirements. The description is too sparse to provide meaningful behavioral transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that is easy to parse. It avoids fluff and gets straight to the point. However, it is somewhat under-specified for the tool's complexity, which slightly lowers the score from a 5.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 3 parameters, no output schema, and no annotations. The description only states the basic action, omitting critical context such as what the result looks like, whether a dataset_id is required, and how the limit applies. This is insufficient for an AI agent to fully understand the tool's behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has only 33% coverage (limit's description), and the description adds no parameter information. It does not explain what 'sql' should contain, how 'dataset_id' is used, or the meaning of 'limit' beyond schema. With low schema coverage and no description compensation, parameter semantics are effectively absent.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Execute a read SQL query through DataProbe.' It has a specific verb ('execute'), a clear resource ('SQL query'), and a scope ('read'), which distinguishes it from siblings like dataprobe_ask (which likely handles natural language) and dataprobe_health/list_datasets.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. It does not mention any prerequisites, exclusions, or compare itself to sibling tools like dataprobe_ask. The only implicit clue is 'SQL query', but there is no explicit context for choosing it over other DataProbe tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• dataprobe_list_datasetsB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It accurately conveys a read-only listing behavior via the word 'List', but it does not disclose whether the operation is expensive, requires authentication, or what the return format looks like. This is a minimal but non-misleading disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single focused sentence with no redundant words. It earns its place and is optimally concise for a simple list operation.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (0 params, no output schema), the description is adequate but incomplete. It does not explain the structure of the returned dataset list or any potential filtering options, so the agent has limited context about what the result will contain.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters and an empty input schema, so the baseline is 4. The description does not need to explain parameter semantics, as there are none to explain.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a verb ('List') and resource ('available DataProbe datasets'), which distinguishes it from sibling tools like dataprobe_health and dataprobe_ask. However, it lacks any scope or detail about what a dataset is, so it is not a full 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. It simply states the function without mentioning prerequisites, exclusions, or context, leaving the agent to infer usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• dataprobe_get_ask_resultA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It only says 'Get', implying a read operation, but fails to describe behavior on invalid or nonexistent query ids, whether results are immediately available or may require polling, or any error handling. The description is too sparse for a tool with zero annotation support.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no redundant content. Every word contributes to the core meaning, making it efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter retrieval tool, the description is adequate but incomplete. It fails to explain the prerequisite that a query_id must first be obtained from dataprobe_ask, and with no output schema, it does not hint at what the returned result contains. The description covers the basic action but lacks integration context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description mentions 'by query id', which provides context that the parameter identifies the ask result, but it does not explain where the id comes from (e.g., from a dataprobe_ask response) or its expected format. With schema coverage at 0%, the description adds minimal value beyond the parameter name itself.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Get') and identifies the resource ('DataProbe ask result') and the key qualifier ('by query id'). This clearly distinguishes it from sibling tools like dataprobe_ask (submission) and dataprobe_query_sql (SQL query).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'by query id' implies the tool should be used after an ask is submitted via dataprobe_ask, but this prerequisite is not explicitly stated. No alternative tools are mentioned, nor are any exclusions given. Basic context is present but not fully articulated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• dataprobe_askA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses a key behavioral trait (polling until the answer is finished), which adds value beyond the schema. However, it omits details about return values, error handling, side effects, or safety, leaving significant behavioral aspects undisclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, front-loaded with the action ('Ask a natural-language data question') and the polling behavior. No redundant words or information; it is appropriately concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema and a polling-centric tool, the description fails to explain what the tool returns after polling finishes (e.g., the answer itself or a status). It also does not address potential timeouts or failure modes. This is adequate for basic invocation but has clear gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, with all parameters described in the input schema (e.g., query, history, max_polls, dataset_id, poll_interval_ms). The description adds no additional parameter-level detail, so the baseline score of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description uses a specific verb 'Ask' and resource 'natural-language data question', and clearly indicates a polling behavior ('poll until the DataProbe answer is finished'). This distinguishes it from sibling tools like dataprobe_get_ask_result, making its purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly conveys when to use this tool: to ask a natural-language question and wait for completion via polling. It does not explicitly mention alternatives or when-not-to-use, but the context is strong enough given sibling names and the stated polling behavior.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• dataprobe_healthA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The disclosure of required credentials (DATAPROBE_ACCESS_TOKEN or username/password) adds operational context. However, it does not describe the response format, whether the call is read-only, or what indicates healthy vs unhealthy status, which would be useful given no annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with no redundancy. It conveys the core action and a key requirement efficiently.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple health check, the description covers purpose and auth, but since there is no output schema, it should ideally mention what the tool returns. This is a noticeable gap that prevents full completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are no parameters, so the description is not expected to document any. The baseline for 0 params is 4, and the description doesn't introduce any parameter-related ambiguity.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Check' and a clear resource 'DataProbe service health', making the tool's purpose unambiguous. It also differentiates from sibling tools that handle datasets, questions, and SQL queries.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is used to verify service health and states authentication prerequisites. However, it does not explicitly discuss when to use this over alternatives, though the distinction is fairly obvious from the name.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.