A/B test significance calculator

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already establish this is a read-only, idempotent, closed-world computation, so the safety profile needs no restating. The description adds genuinely useful behavioral context beyond that: which test variants are run, that the z-test is pooled, and that chi-square matches R's prop.test with and without Yates — telling the agent what methodology to expect. It does not cover edge behavior (zero-conversion groups, tiny samples), keeping it at 4.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the question the tool answers, then a tight enumeration of outputs. It is dense and single-sentence but every clause adds information; no filler or restatement of the name.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema, so the description must convey what comes back, and it does so thoroughly by listing each computed statistic. For a pure calculator with rich annotations the definition is nearly sufficient; only the response shape and degenerate-input behavior are unstated.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 43%, with the four required parameters (control/variant visitors and conversions) carrying no descriptions. The phrase 'From visitors and conversions per group' partially compensates by mapping them, and the three optional params are fully documented in the schema. Baseline 3 is appropriate given the coverage gap is only partly closed.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

Opens with a specific question the tool answers and enumerates the exact statistics computed (pooled two-proportion z-test, Wald CI, chi-square with/without Yates, Fisher exact, SRM). That is a concrete verb+resource and scope. It does not explicitly say how it differs from the sibling fisher_exact_test, which it partially overlaps with, so it stops short of 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The leading question implies when the tool applies (you have a finished A/B result and want a significance verdict), but there is no explicit when-to-use / when-not guidance and no routing to siblings such as fisher_exact_test or check_peeking_risk, which an agent would plausibly confuse with this one.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.