A/B Test Sample Size Calculator MCP

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already establish that this is a safe read-only, idempotent, non-destructive operation. Beyond that, the description adds meaningful methodological transparency: the pooled/unpooled normal formula, comparison to power.prop.test, and reconciliation with Evan Miller's calculator output. It does not explain output structure in detail, but the statistical behavior is well disclosed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is front-loaded with the core purpose and then layers in method and output context. It is dense and slightly run-on, but every clause carries relevant information, so it is appropriately concise for a statistical calculator.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema, so the description must carry some return-value context. It does so by stating that the tool reports exact per-variant sample size and the corresponding Evan Miller figure, though it does not describe the full output shape or units in detail. For a 9-parameter calculator, it is mostly complete but leaves some output clarity implicit.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so all nine parameters are already documented in the input schema. The description mentions the same parameter concepts at a high level, including baseline rate, MDE type, alpha, power, sidedness, variants, Bonferroni, and allocation, but it does not add syntax, edge-case, or default-value detail beyond what the schema provides.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource: calculating exact per-variant sample size for an A/B test on a conversion rate using two independent proportions. It clearly distinguishes itself from generic statistical tools by naming the method, but it does not explicitly contrast itself with sibling tools such as estimate_test_duration.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies when the tool is appropriate by listing supported scenarios: relative or absolute MDE, multiple variants, Bonferroni correction, and unequal allocation. However, it gives no explicit guidance about when to use this instead of estimate_test_duration or other siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover the safety profile (read-only, idempotent, non-destructive), so the description adds useful behavioral context by naming the false-positive-rate calculation, an example value, and the boundary families. However, it does not reconcile the 'up to 10 equally spaced looks' statement with the schema's maximum of 50, leaving a behavioral limit ambiguous.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is front-loaded and contains no filler, but it packs multiple outputs and a constraint into one dense sentence, reducing scanability slightly.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a two-parameter read-only computation with no output schema, the description names the returned quantities and boundary types, so an agent knows what to expect. It still leaves the output structure and the 10-vs-50 looks limit ambiguous.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, and the schema already documents alpha and looks. The description reinforces the meaning of repeated looks and gives an example at alpha 0.05, but the phrase 'up to 10 equally spaced looks' conflicts with the schema's maximum of 50 for looks, which can confuse parameter validity.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description specifies the computed quantities: the real false positive rate under repeated checks and Pocock/O'Brien-Fleming sequential boundaries. It is clearly about peeking/sequential-testing risk, but it does not explicitly distinguish this from sibling tools such as test_ab_significance or calculate_sample_size.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

No sentence states when to use this tool versus the sibling significance or sample-size tools. The peeking-risk framing implies a planning or repeated-checking use case, but the agent must infer that and may confuse it with test_ab_significance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover the safety profile (readOnly, idempotent, non-destructive, closed-world), so the description's job is to add substance beyond that — and it does, disclosing that the tool returns both exact days and a recommended whole-week figure rounded up to absorb weekday effects. That return-shape context matters especially since there is no output schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

A single front-loaded sentence with a clear subject (days required), the two input modes, and the output guarantee — no filler and nothing buried. The parenthetical about calculate_sample_size is load-bearing rather than decorative.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a read-only, closed-world estimator with 9 parameters and no output schema, the description gives enough to call it correctly and to know what comes back. Minor gaps remain: behavior when both sample_size_per_variant and baseline_rate/mde are provided, and whether mde_type interacts with the baseline path.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 78%, so the schema already documents most parameters, but the description adds the grouping logic an agent needs: which parameters form the sample-size branch, which form the baseline-rate branch, and that traffic_share excludes the non-entered portion of daily visitors. It does not clarify precedence if both branches are supplied, leaving one ambiguity.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific computation and its unit ('How many days an A/B test must run') plus the two ways to feed it, which is enough to distinguish it from calculate_sample_size (sample size) and test_ab_significance (result analysis). The parenthetical cross-reference to calculate_sample_size makes the boundary with that sibling explicit.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

Clearly describes the two entry paths — known 'sample size per variant' vs 'baseline rate and MDE' — and names the inputs required alongside them (daily visitors, traffic share). It stops short of stating when not to use it or which of the sibling tools handles the adjacent question, so it is clear context rather than full routing guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly/idempotent/non-destructive, so the safety bar is met; the description adds real behavioral context beyond them — the network algorithm used for large tables and the fact that Pearson chi-square plus a low-expected-count tally are returned alongside the exact p-value. It does not describe error behavior for invalid tables (e.g. fewer than 2 rows), a minor gap.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Three sentences, front-loaded with the core purpose and then branching by input shape, followed by the secondary return values. No filler and nothing repeated from the schema or annotations.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema, and the description compensates by naming the returned quantities (exact p-value, Pearson chi-square, count of low-expected cells). For a two-parameter pure-computation tool, an agent has everything needed to call it and interpret the result.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so both 'table' and 'alternative' are already documented in structured data, including the enum and its 2x2-only restriction. The description's 'two-sided, greater or less, as R fisher.test' adds only a mild semantic anchor, which fits the baseline-3 case where the schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource: exact p-value for a contingency table of counts, then splits the behavior by table shape (2x2 → Fisher exact, r x c → Fisher-Freeman-Halton). An agent can tell immediately what statistical output this produces and for which input shape.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The 2x2 vs larger-table routing rule is explicit, and the note that expected counts below 5 signal 'chi-square approximation is unsafe' gives the agent a decision criterion for choosing this over approximate methods. It stops short of naming a specific sibling (e.g. test_ab_significance) as an alternative, so it is clear context without explicit exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already establish this is a read-only, idempotent, closed-world computation, so the safety profile needs no restating. The description adds genuinely useful behavioral context beyond that: which test variants are run, that the z-test is pooled, and that chi-square matches R's prop.test with and without Yates — telling the agent what methodology to expect. It does not cover edge behavior (zero-conversion groups, tiny samples), keeping it at 4.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the question the tool answers, then a tight enumeration of outputs. It is dense and single-sentence but every clause adds information; no filler or restatement of the name.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema, so the description must convey what comes back, and it does so thoroughly by listing each computed statistic. For a pure calculator with rich annotations the definition is nearly sufficient; only the response shape and degenerate-input behavior are unstated.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 43%, with the four required parameters (control/variant visitors and conversions) carrying no descriptions. The phrase 'From visitors and conversions per group' partially compensates by mapping them, and the three optional params are fully documented in the schema. Baseline 3 is appropriate given the coverage gap is only partly closed.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

Opens with a specific question the tool answers and enumerates the exact statistics computed (pooled two-proportion z-test, Wald CI, chi-square with/without Yates, Fisher exact, SRM). That is a concrete verb+resource and scope. It does not explicitly say how it differs from the sibling fisher_exact_test, which it partially overlaps with, so it stops short of 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The leading question implies when the tool applies (you have a finished A/B result and want a significance verdict), but there is no explicit when-to-use / when-not guidance and no routing to siblings such as fisher_exact_test or check_peeking_risk, which an agent would plausibly confuse with this one.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.