Server Quality Checklist

Tool Scores

• mdx_runB
  3.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must disclose behavioral traits. It does so by clarifying read-only constraint and that it returns a single cell value. However, it lacks details on error handling, performance, or what happens if the query returns multiple cells.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, very concise with no superfluous words. It efficiently conveys key information, though it could be expanded with an example without losing conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations, no output schema, and only one parameter with no description, the description is insufficient for full context. It omits important details like error messages, syntax specifics, or time limits, leaving significant gaps for an AI agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, and the description does not elaborate on the 'mdx' parameter beyond 'a provided MDX query'. No format, constraints, or examples are given, leaving the agent without guidance on how to construct the query.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the tool executes an MDX query, specifies its read-only nature (SELECT/WITH only), and mentions returning a cell value. However, it does not explicitly differentiate from sibling tools like mdx_ask.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for read-only queries by stating 'read-only; SELECT/WITH only', but does not provide when to use this tool over alternatives like mdx_ask or mdx_explain. No guidance on when not to use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• mdx_explainB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden for behavioral disclosure. It only states the tool explains the query but omits details about side effects, permissions, rate limits, or whether the explanation is static or AI-generated. This leaves the agent uncertain about behavioral traits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that conveys the core purpose efficiently. It could be slightly more structured (e.g., listing what the explanation includes), but it remains concise without extraneous words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has one parameter and an output schema (which documents return values), the description addresses the primary purpose. However, missing usage guidelines and behavioral transparency leave gaps. For a simple tool, it is adequate but not fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% for the single 'mdx' parameter. The description identifies it as an MDX query but does not specify format, constraints, or examples. The parameter's semantics are implied rather than explicitly defined, which is insufficient for an agent to construct valid input.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'explain', the resource 'MDX query', and the output 'plain language covering measures, slices, and returns'. It effectively differentiates from sibling tools like mdx_run and mdx_ask by focusing on explanation rather than execution or questioning.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is used for explaining MDX queries, but does not explicitly state when not to use it or mention alternatives. Given sibling tools, additional guidance on when to choose mdx_explain over mdx_ask or mdx_introspect would be helpful.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• mdx_introspectA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It states the tool 'returns' data, indicating a read-only operation, but does not disclose potential side effects, authentication needs, or rate limits. The description is minimal but not misleading.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that immediately conveys the tool's purpose and output. Every word adds value, with no fluff or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that the tool has no parameters and an output schema exists (presumably defining the return structure), the description is adequate. It identifies what is returned (measures, dimensions, hierarchies) and hints at usage ('for grounding'). Could be more complete by explicitly stating that the output schema provides full detail.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters and schema coverage is 100%, so the description does not need to add parameter meaning. It simply confirms the tool takes no arguments, which is sufficient. Baseline 4 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns the cube's 'skills' block consisting of measures, dimensions, and hierarchies. This distinguishes it from siblings like mdx_ask (which likely asks questions) and mdx_run (which executes queries), making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'for grounding' implies use in providing context for other tools, but there is no explicit guidance on when to use versus alternatives (e.g., mdx_explain). No exclusions or prerequisites are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• mdx_askA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Despite no annotations, the description comprehensively explains the internal behavior: introspecting the cube, generating k candidate queries, executing them read-only, and returning a verdict. It also discloses the return structure and the abstain/clarify outcomes, leaving little ambiguity.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with three sentences: a clear purpose statement, a detailed process overview, and a cautionary note. Every sentence adds value without fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity, the description covers input (question, k), process (generation, execution, verification), and output format (status, value, etc.). No output schema is needed because the structure is described. It fully equips the agent to use the tool appropriately.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema coverage, the description adds meaning by explaining that 'k' controls the number of candidate MDX queries generated. The 'question' parameter is obvious from context. It doesn't specify the default behavior when k is null, but the schema shows it's null.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: to answer natural-language questions with verified MDX results. It distinguishes itself by highlighting the self-consistency verdict mechanism, which sets it apart from siblings like mdx_run.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides context on when to use the tool (for natural-language queries needing verification) and cautions that it never guesses when candidates disagree. However, it does not explicitly state when not to use or mention alternatives like mdx_run for direct MDX execution.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.