Server Quality Checklist

Tool Scores

• eval_classifier_robustnessC
  2.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description should disclose behavioral traits. It mentions output format but does not state whether the tool is read-only, requires authentication, or has side effects. The analysis nature suggests safety, but it's not explicit.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise at two sentences with no wasted words. However, front-loading is acceptable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 8 parameters, no annotations, and an output schema, the description is notably incomplete. It glosses over evaluation stages, parameter roles, and output specifics.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% (no parameter descriptions), and the tool description adds no meaning beyond parameter names. It does not explain 'query', 'topic', 'obfuscation', or other parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the tool processes a query through evaluation stages and returns analysis, which is moderately clear. However, it is vague about what 'configured evaluation stages' entails and does not differentiate from sibling tools like eval_decompose_query or eval_obfuscation_evasion.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. There is no mention of context, prerequisites, or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• eval_decompose_queryC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description must fully disclose behavioral traits. It only states that the query is split into parts based on topic boundaries. It does not mention side effects, whether the operation is read-only, how the splitting algorithm works, or any constraints. This is insufficient for a behavioral understanding.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise at two sentences, with no redundant information. The first sentence states the core purpose, and the second adds detail. It is front-loaded and efficient, though some might argue it is too brief given the missing parameter details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the presence of an output schema (not shown), the description does not need to explain return values, but it still lacks information about parameters, usage context, and behavioral traits. The tool has three parameters with no description, which makes it incomplete for effective use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage for parameters, so the description must compensate. However, the description does not mention any of the three parameters (query, max_pieces, auto_split) or their semantics. It only describes the high-level function, providing no meaning beyond the parameter names in the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool splits a query into conceptual sub-components based on topic boundaries. It is specific about the action and resource, and while it doesn't explicitly differentiate from siblings, the sibling tools have distinct purposes (robustness, evasion, status), so confusion is unlikely.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. There is no mention of prerequisites, contexts, or scenarios where the tool is appropriate or inappropriate. The description lacks any exclusions or alternative tool names.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• eval_obfuscation_evasionB
  3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description states the tool applies character substitutions and reports patterns, indicating a non-destructive analysis. However, it does not disclose specifics like scope of patterns, required permissions, or edge cases. With no annotations, the description carries the full burden but gives only basic transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise: two sentences covering the action and mechanism. It front-loads the purpose. However, it could be more structured (e.g., separate 'when to use' section). Still, every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 2 parameters and no annotations, the description is insufficient for an agent to use it correctly. It lacks details on input format, level options, and output nature (though output schema exists). For a simple analysis tool, it meets the bare minimum but leaves gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%. The description does not explain the 'text' or 'level' parameters. 'Configurable character substitutions' vaguely hints at 'level', but does not describe expected values or effect. The description adds no semantic value beyond what the schema provides, which is minimal.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: analyzing character-level text transformations and reporting detected patterns. The verb 'analyze' paired with the resource 'character-level text transformations' is specific. Sibling tools have distinct purposes (classifier robustness, query decomposition, status), so this tool is well-differentiated.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus alternatives. The description implies it's for detecting obfuscation/evasion, but does not provide when-not-to-use or mention sibling tools. An agent would need to infer usage context from the name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• eval_statusA
  3.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, and the description only states the tool 'check's configuration and status, which is a generic read behavior. No additional behavioral context (e.g., auth needs, side effects) is disclosed beyond the obvious.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence that conveys the purpose without any redundant or extraneous information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (no parameters, output schema present), the description adequately covers its purpose. No additional details are necessary for this self-contained status check.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are no parameters, so the description does not need to add meaning beyond the schema. A baseline score of 4 is appropriate as there is no information to provide.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'check' with resources 'server configuration and connection status', clearly distinguishing it from sibling tools like eval_classifier_robustness which have different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this tool is for checking status, and sibling names suggest distinct use cases (e.g., robustness, decomposition), but no explicit guidance on when to use or avoid using this tool is given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.