ab_test

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description fully carries the behavioral burden. It discloses the key behavior: peeking is safe and does not inflate false-positive rates. This is critical for correct usage. However, it does not mention other behaviors like mutability or side effects, which are not relevant here.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with the core question, and immediately providing the essential statistical property. Every sentence adds value without redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool lacks an output schema and the description does not explain what the tool returns (e.g., recommendation, p-value, winner). Given 7 parameters and no output schema, the description is incomplete for an AI agent to fully understand the tool's behavior.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema covers 71% of parameters with descriptions. The description adds no additional meaning to parameters beyond what the schema provides. For the 29% not covered (a_label, b_label), the description does not compensate. Baseline of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly asks 'Can you call a winner on an A/B test yet?' which directly states the tool's purpose. It specifies the methodology (confidence sequences) and distinguishes from sibling tools like forecast_next or did_it_change.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly states that the tool uses anytime-valid confidence sequences and that it is safe to run after every observation without inflating false-positive rates. It contrasts with t-test/z-test, providing clear usage guidance, though no direct comparison to siblings is given.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.