Server Quality Checklist

Tool Scores

• tailtest_scenario_planA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full behavioral burden. It details the returned content but does not disclose side effects, permissions, or whether the operation is read-only. It is adequate but lacks depth on behavioral traits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with purpose, and efficiently lists all components without redundancy or unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description lists the fields returned but does not specify the output format (e.g., JSON structure) or address error cases. Given no output schema, the agent may need to infer the structure, but the listing is helpful.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema descriptions already cover file_path and project_root with 100% coverage. The description does not add additional meaning beyond what the schema provides, so it is at baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns 'structured scaffolding' for writing a scenario plan, listing specific components like language, framework, depth, R15 adversarial count, baseline scenarios, test path, and instructions. It distinguishes from sibling tools by focusing on scaffolding generation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides context that the scaffolding is used before composing the actual scenario plan, but it does not explicitly exclude other uses or compare with alternatives like other tailtest tools. The usage is implied.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• tailtest_pick_templateA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full responsibility. It discloses the fallback behavior ('Returns just language baseline when no framework matches') and describes the output components. However, it does not mention error conditions or side effects, but given the read-only nature, this is acceptable.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, starts with the verb 'Return', and contains zero redundant information. Every phrase contributes to understanding the tool's function.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given two parameters and no output schema, the description adequately explains the return value and the fallback. It lacks mention of error cases or supported languages, but is otherwise complete for a straightforward read tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, but the parameter descriptions are minimal. The tool description adds value by explaining that the output is determined by the file_path and project_root, and contextually connects parameters to the template resolution. This goes beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it returns a framework R2 template for a given source file, listing specific components (language baseline scenarios, framework baseline scenarios, etc.). This is a specific verb+resource and differentiates from sibling tools like tailtest_classify_failures.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for picking a template for a source file but does not explicitly state when to use this tool versus alternatives. It lacks guidance on when not to use or naming sibling tools for comparison.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• tailtest_classify_failuresA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description must fully disclose behavior. It details the return fields (type, reason, test name, etc.) and mentions the agent can verify/override. It does not mention side effects, which seems acceptable for a read-only classification. Could note that no state is modified.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, each adding essential information: what it does, what it returns, and how the agent interacts. No unnecessary words, front-loaded with the main action. Highly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Without an output schema, the description fully covers the return value details (fields and categories). It explains the heuristic and agent override capability. No gaps are apparent for the tool's complexity. Enough for an agent to correctly invoke and interpret results.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with clear descriptions for both parameters. The description confirms the 'runner' parameter defaults to pytest, which is already in the schema. It does not add significant new meaning beyond the schema, but it does tie parameters to the overall classification context, earning a baseline score.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: parsing runner output and applying heuristic R12 classification. It specifies the exact resources (runner output, runner) and outputs (failure types, categories). The tool distinguishes well from siblings (none involve classification).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context: use when you have test runner output to classify. It does not explicitly exclude scenarios or name alternatives, but sibling tools are clearly different (template, ping, plan, setup), so confusion is low. A brief 'when not to use' would improve it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• tailtest_pingA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, but the description honestly discloses the read-only nature and return of server version. No hidden behaviors.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single concise sentence with no redundancy, efficiently conveying the tool's purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple ping tool with no parameters or output schema, the description fully covers its behavior and return value.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters exist; schema coverage is 100%. Baseline for 0 params is 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it is a health check that returns server version and confirms reachability, distinguishing it from sibling tools like classification or setup.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Usage is implied as a health check, but no explicit guidance on when to use vs alternatives or when not to use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• tailtest_setupA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description fully carries the burden. It discloses all major effects: file creation (rules, workflows, memory bank, configs), non-overwriting of existing files, and the reload warning. No contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise (3-4 sentences) yet packed with information. It front-loads the main purpose and each sentence serves a clear role without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description partially explains the return value (structured report with reload warning) but omits details about report fields or structure. For a tool that modifies multiple files, this is a minor gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, but the description adds valuable context beyond the schema by explaining the implications of the mode parameter (manual vs. auto) and what triggers each. This adds semantic richness.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose as the bootstrap entry point for tailtest, listing specific actions like detecting language, writing rule packs, and initializing configs. It is distinctly different from sibling tools (classify, pick, ping, scenario plan).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage as the first step in tailtest setup, but it does not explicitly state when to use or avoid it. It provides clear context but lacks explicit exclusions or alternative tool guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.