Server Quality Checklist

Tool Scores

• simulate_network_dropA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses the core behavior (aborting requests after a delay) but omits the fallback verification feature (fallback_selector) and any potential side effects. This is a clear gap, as the schema indicates the tool can check for a fallback UI, which is an important behavioral trait.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core action and followed by a concrete use-case example. There is no redundant information or irrelevant filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description fails to mention the fallback checking behavior implied by fallback_selector and does not describe the overall workflow (load, drop, verify). With six parameters, no output schema, and no annotations, the description is too sparse to fully guide an agent on what the tool does end-to-end.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema covers all parameters with descriptions (100% coverage), so the description doesn't need to repeat them. It adds minimal value beyond the schema, though the example scenario helps illustrate the purpose of intercept_pattern and drop_after_ms. Baseline 3 is appropriate given the schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('aborts') and resource ('requests matching a pattern') and clearly states the behavior in the first sentence. It is easily distinguished from sibling tools like inject_latency (delays only) and block_resources (blocks immediately) by describing a delayed mid-flight drop.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The second sentence provides explicit context: 'Use to answer: what happens if the network drops after the order request is sent but before the response arrives?' This clearly indicates when the tool is appropriate, but it does not mention alternatives or when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• block_resourcesA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses the primary behavior (blocking requests) and implies a post-block content check via core_content_selector, but it doesn't disclose the return format, how success/failure is signaled, or whether the block is temporary and scoped to the test. The description adds some behavioral context but not enough for a tool with no annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the main action, and the second sentence provides a practical use case. Every phrase earns its place, with no redundancy or unnecessary detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description explains the 'why' and gives a clear scenario, but it doesn't specify what the tool returns or how it communicates success/failure after blocking requests. Given that there is no output schema and no annotations, the description should do more to describe execution outcomes. However, the schema covers parameter constraints well, and the tool's purpose is well-articulated, so it's adequate but incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, so the baseline is 3. The description adds high-level context (e.g., simulating third-party outages) but doesn't explain parameter interactions or nuances beyond what the schema already provides. The schema's own descriptions, including examples for block_patterns, are already detailed, so the description adds marginal value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Blocks') and resource ('requests to specified URL patterns'), and immediately distinguishes itself from siblings by focusing on selectively blocking third-party resources like analytics/CDNs. It also provides a concrete use-case scenario, making the tool's purpose immediately clear.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use the tool: 'useful for simulating third-party outages' and gives a concrete question it can answer. It doesn't explicitly mention when not to use it or name alternative tools, but the context is clear enough to guide selection among the sibling chaos tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• inject_latencyA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It states the action (adding artificial delay) and its purpose (simulating slow networks/overloaded APIs), but does not disclose details like reversibility, session scope, cleanup, or impact on subsequent requests. It is not misleading, but leaves out behavioral nuances expected for a network-interception tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core action, followed by concrete use-case questions. It is concise, zero waste, and every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 6 parameters, a nested viewport object, and no output schema, the description provides essential purpose and use cases but does not explain the interplay between latency_ms and jitter_ms, the role of loading_selector, or potential side effects. It is adequate for selection but not fully complete for invocation without relying on schema details.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the schema already documents all six parameters thoroughly. The description does not add parameter-specific meaning beyond mentioning 'URL pattern' (which relates to intercept_pattern). Baseline 3 is appropriate since the schema does the heavy lifting, and the description adds minimal unique value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function with a specific verb ('Adds artificial delay') and resource ('requests matching a URL pattern'). It distinguishes itself from sibling tools like simulate_api_failure or block_resources by focusing on latency injection rather than errors or blocking.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides concrete use cases ('Does the app show loading states when the API takes 3 seconds? Does it time out gracefully?'), giving clear context for when to use this tool. It does not explicitly mention alternatives or exclusions, but the use cases are specific enough to guide an agent.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• inject_response_corruptionA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden of behavioral disclosure. It explains the core behavior (intercepting and corrupting responses) but does not mention potential side effects, cleanup, or whether the corruption affects subsequent requests. This is adequate but not comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the purpose and followed by a concrete use-case question. Every word earns its place with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers purpose and usage, and the schema handles parameter semantics. However, there is no output schema and the description does not explain return values or how the fallback_selector is used, leaving some ambiguity about what the agent should expect after invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents all parameters. The description reinforces the conceptual meaning of corruption types but adds little beyond the schema's own parameter descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: intercepting requests and returning malformed or corrupted responses. It also distinguishes this tool from siblings by specifying protocol-level corruption such as malformed JSON, content-length lies, and truncated payloads.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides an explicit use case ('Use to answer: does the app handle...?'), giving clear context for when to apply the tool. It does not explicitly name alternatives or exclusions, but the guidance is sufficiently clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• simulate_api_failureA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the disclosure burden. It explains the core behavior (intercept, return error, navigate, check fallback) but does not disclose potential side effects (e.g., whether the interception persists, whether the browser session is cleaned up, or if the tool is safe for production use). This is basic but not exhaustive transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the main action, and free of filler. Every sentence earns its place: the first defines the mechanism, the second defines the verification step and a concrete use case.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is complex (7 params, no output schema) and the description does not explain what the tool returns or how to interpret the result (e.g., whether it returns a boolean or a report). The parameter schema is rich, and the description covers the high-level flow, but the missing return-value information leaves a gap in completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description adds the example 'payment API returns 503' which clarifies the intended values for status_code and intercept_pattern, but it does not elaborate on parameter semantics beyond what the schema already provides. No extra value to justify a higher score.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific action ('Intercepts API requests matching a URL pattern and makes them return an error status code') followed by the testing flow ('Navigates to the page and checks if a fallback UI element appears'). It distinguishes itself from siblings like inject_latency or block_resources by focusing on fault injection plus UI verification.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives a clear use case: 'Use to answer: does the app show a proper error state when the payment API returns 503?' This implies when the tool should be used. However, it does not explicitly mention when not to use it or name alternative tools, so it falls short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• simulate_stateful_failureA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses the core stateful behavior (fails first N, then succeeds), the error status, and the success payload concept. It does not mention cleanup/reset or how the tool handles multiple navigations, but it provides substantive behavioral context beyond the schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences that lead with the action and then state the use case. No redundant text; every sentence contributes.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description explains the main behavior and use case but lacks details about what the tool returns or how fallback_selector integrates into the test. Given no output schema, the agent would benefit from knowing the result format. The parameter semantics for nested viewport are left to the schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description implicitly explains failure_count and http_status ('fails the first N requests with an error status'), but does not add detail for wait_ms, viewport, or fallback_selector. It adds minimal value beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool intercepts requests, fails the first N with an error status, then allows success. This specific behavior distinguishes it from sibling tools like simulate_api_failure or inject_latency, which likely don't implement the stateful failure recovery pattern.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly frames the tool for testing retry/recovery logic with the example 'does the app retry after a 503 and recover when the service comes back?' This gives clear context for use, though it doesn't explicitly name when not to use or compare to alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• trigger_system_network_errorA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must disclose behavior. It states the tool aborts matching requests and simulates specific OS-level failures, which is transparent about the primary effect. However, it omits potential side effects (e.g., whether the page becomes non-interactive, whether the intervention is temporary) and does not warn about testing environments.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences; the first states the action and simulation type, the second gives a concrete test question. No redundant content or padding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 6 params and no output schema; the description conveys the core purpose but does not outline the end-to-end workflow (e.g., navigating to the URL, waiting, and evaluating the fallback selector). It gives the 'why' but not the full 'how', which the agent must infer from the schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema describes all parameters fully (100% coverage), so the baseline is 3. The description only mentions 'URL pattern' in passing and does not add details about format, units, or interplay between parameters beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Aborts') and identifies the resource ('requests matching a URL pattern') and distinguishes it by focusing on system-level network error codes that simulate OS-level failures, unlike siblings that simulate API or latency issues. The use-case question further clarifies unique purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly frames when to use it: to test app recovery from DNS resolution failures or OS connection rejections. It implies this tool is for OS-level network failures but does not name alternatives or state when not to use it, so no exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• assert_chaos_handledA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden and does well: it discloses the injection of chaos status, the evaluation dimensions (fallback UI, unhandled exceptions, survival), and implies a structured verdict. It adds context beyond the schema, though it stops short of detailing the exact verdict shape or side effects like browser navigation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two focused sentences: the first describes the mechanism and verdict, the second gives a direct use case. No redundancy, front-loaded with the most critical information, and every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has no output schema, so the description compensates by summarizing the structured verdict (fallback UI, unhandled exceptions, survival). It covers the central use case and key behavior. It leaves some details—like the exact return fields—to inference, but for an assertion tool this is reasonably complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% parameter coverage with detailed descriptions, including defaults and logic for expected_fallback_selector. The tool description adds minimal extra meaning beyond a high-level flow, so it meets the baseline but does not significantly improve parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific action: injects a chaos HTTP status into matching requests, then returns a verdict on resilience. It distinguishes from sibling tools by emphasizing assertion/evaluation rather than just failure injection, naming the verdict components (fallback UI, unhandled JS exceptions, survival).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly tells when to use: 'Use to answer: is the app chaos-resilient — does it show a recovery UI without throwing exceptions?' This gives clear context, though it does not explicitly contrast with sibling tools or state when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.