How do I use eval-sample-size?

1. Click on "Deploy Server". 2. Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state. 3. In the chat, type @ followed by the MCP server name and your instructions, e.g., "@eval-sample-size how many test cases do I need to tell 82% from 78% accuracy?" That's it! The server will respond to your query, and you can continue using it as needed. Here is a step-by-step guide with screenshots.

eval-sample-size

by fayehall-ai

Overview Schema Source Related Servers Discussions Admin

Python

Remote

sample-size-mcp

How many test cases do you need to trust that model A is really better than model B?

Usually more than you'd think: telling 82% from 78% takes about 1,500 cases if the models' errors are unrelated. If both models run on the same test set, and they usually make the same mistakes, the number can drop to about 500. This tool does the math. You can use it as a CLI or as an MCP server, so an agent can check an eval before trusting it.

CLI

uv run evalsize n 0.82 0.78                     # cases needed (same test set for both models)
uv run evalsize n 0.82 0.78 --discordance 0.10  # ...when the models disagree on 10% of cases
uv run evalsize n 0.82 0.78 --unpaired          # each model gets its own sample
uv run evalsize gap 500 0.78                    # smallest win 500 cases can detect
uv run evalsize compare 500 --a-only 41 --b-only 22 --both 360   # is this result real?

$ uv run evalsize n 0.82 0.78 --discordance 0.10
489 test cases (shared by both models)
  to detect 82.0% vs 78.0% (gap 4.0%) at alpha=0.05, power=80%
  discordance 10.0% (given); ~49 cases will carry the signal

$ uv run evalsize compare 500 --a-only 41 --b-only 22 --both 360
A 80.2% vs B 76.4%: gap +3.8%, 95% CI [+0.7%, +6.9%], p=0.0226 -> SIGNIFICANT

All commands accept --alpha (false-positive rate, default 0.05) and --power (the chance of detecting a real gap, default 0.8).

Related MCP server: evalgate

MCP server

{
  "mcpServers": {
    "eval-sample-size": {
      "command": "uvx",
      "args": ["--from", "git+https://github.com/fayehall-ai/sample-size-mcp", "sample-size-mcp"]
    }
  }
}

Or with Claude Code: claude mcp add eval-sample-size -- uvx --from git+https://github.com/fayehall-ai/sample-size-mcp sample-size-mcp

Tools: eval_sample_size, eval_min_detectable_gap, eval_compare_paired, eval_compare_unpaired.

The statistics

Paired (the default) means both models are scored on the same cases. Cases where both models are right, or both wrong, tell you nothing about which model is better. Only the cases where they disagree count, so the required sample size depends on discordance: the fraction of cases where exactly one model is right. This tool uses McNemar's test, with sample size from Connor (1987):

n = (z_{α/2}·√ψ + z_β·√(ψ − δ²))² / δ²       ψ = discordance, δ = accuracy gap

If you don't pass a discordance, the tool assumes the models' errors are independent, ψ = a(1−b) + b(1−a). That's the worst realistic case, and it needs about as many cases as the unpaired design. Real models usually fail on the same hard cases, so the true discordance is lower and you need fewer cases. Run a small pilot (~100 cases), measure how often the models disagree, and pass that number. This is usually the biggest lever you have.

Unpaired means each model is scored on its own random sample. This uses the two-proportion z-test (Fleiss formula, no continuity correction), and n is the number of cases per model.

Checking results: compare / eval_compare_paired uses the exact McNemar (binomial) test, with a Wald confidence interval on the gap.

Tests include a Monte Carlo check that the recommended n actually gives about 80% power. Run them with uv run pytest.

Caveats

Every test case counts as independent. If your cases come in clusters (several prompts per document, or several samples per prompt), you need more cases than this says.
Each test is a single comparison. If you compare many models or checkpoints, divide alpha by the number of comparisons (Bonferroni correction).
For accuracies close to 0 or 1, or very small samples, the normal approximation becomes rough.

Available Tools

4 tools

eval_compare_pairedA

Is A's win over B on a shared test set statistically real?

n: total cases; a_only: A right & B wrong; b_only: B right & A wrong;
both: both right. Returns exact McNemar p-value and a CI on the gap.

Name	Required	Description	Default
`n`	Yes
`both`	Yes
`alpha`	No
`a_only`	Yes
`b_only`	Yes

TDQS

A3.5/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations exist, so the description carries the full burden. It usefully discloses that the p-value is exact and that a CI on the gap is returned, but says nothing about what alpha controls, error conditions (e.g., inconsistent n), or determinism.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Very compact and front-loaded: the motivating question comes first, then the count semantics, then the return values. The question form is slightly indirect as a purpose statement but costs almost nothing.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a computation tool with no annotations and no output schema, the description covers inputs and outputs reasonably but omits the alpha parameter and any statement of result shape (statistic vs CI bounds). An agent would still need to guess how alpha changes the output.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% and all titles are just field names, so the description does real work: n, a_only, b_only and both are defined precisely in terms of correctness outcomes. Only alpha is left undocumented, and the relationship between n and the three counts is not stated.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description frames the tool as an exact McNemar test on a shared test set, which is a specific verb+resource (paired comparison of A vs B). 'Shared test set' implicitly separates it from eval_compare_unpaired, though the sibling is never named.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the pairing framing and the 'A's win over B' scenario, but there is no explicit when-to-use/when-not guidance and no pointer to eval_compare_unpaired or eval_sample_size as alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

eval_compare_unpairedC

Is A's win over B real when each was scored on a different sample? Two-proportion z-test with a CI on the gap.

Name	Required	Description	Default
`n_a`	Yes
`n_b`	Yes
`alpha`	No
`correct_a`	Yes
`correct_b`	Yes

TDQS

C2.9/5.0

Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations exist, so the description carries the full burden. It discloses the method (z-test, CI on the gap), which is useful, but says nothing about assumptions such as independence or minimum sample size, whether a continuity correction is applied, what fields come back (p-value? CI bounds?), or how alpha affects results. For a statistical inference tool this leaves significant behavioral gaps.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two short sentences with no filler, and the motivating question is front-loaded before the method statement. The interrogative opening is slightly indirect but functionally efficient.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With 5 undocumented parameters, no annotations, and no output schema, the description does not do enough: an agent cannot tell from it whether correct_a is a count out of n_a, how alpha is used, or what the response contains. The statistical method alone does not compensate for the missing parameter and return-value context.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% and the parameter titles ('N A', 'Correct A', 'Alpha') are ambiguous on their own — 'correct_a' could be a count or a rate. The description's A/B framing gives a weak hint about the a/b parameter grouping but never explains n_a, correct_a, or what alpha controls.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific statistical procedure (two-proportion z-test) and the resource being compared (A vs B on independent samples), plus a CI on the gap. The phrase 'each was scored on a different sample' implicitly separates it from the paired sibling, though it never names that sibling explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

'When each was scored on a different sample' implies the condition that selects this tool over eval_compare_paired, but no alternative is named and no exclusions or prerequisites are stated. Usage is inferable rather than guided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

eval_min_detectable_gapC

Smallest accuracy improvement over baseline_acc that an eval with n test cases can reliably detect. Use this to sanity-check an existing eval set.

Name	Required	Description	Default
`n`	Yes
`alpha`	No
`power`	No
`paired`	No
`discordance`	No
`baseline_acc`	Yes

TDQS

C2.9/5.0

Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations exist, so the description carries the full burden, and it discloses almost nothing behavioral: not that this is a pure read-only computation, not the statistical assumptions baked into the defaults (alpha=0.05, power=0.8, paired test), and not how the result is expressed (accuracy points vs. proportion). Only the headline behavior is stated.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tight sentences with the definition front-loaded and the usage hint second; nothing is padded. The only cost is that brevity here is partly under-specification rather than efficiency.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema, no annotations, and 0% schema coverage for 6 parameters means the description must do the heavy lifting, and it does not: return-value semantics, the paired/discordance interaction, and the effect of alpha/power are all unaddressed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% across 6 parameters, and the description only touches the two required ones (n, baseline_acc) by name. The statistically consequential parameters — alpha, power, paired, discordance — are completely unexplained, leaving the agent to guess at semantics that strongly affect the answer.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Names a specific computed quantity (minimum detectable accuracy improvement) in terms of two named inputs, which is precise enough for an agent to recognize the tool's role in an eval-design toolkit. It does not explicitly contrast itself with eval_sample_size or the compare_* siblings, so the differentiation is left implicit.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

'Use this to sanity-check an existing eval set' gives one concrete scenario, which is more than nothing. But there is no statement of when NOT to use it (e.g., to plan a new eval, use eval_sample_size instead) and no mention of the sibling tools at all.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

eval_sample_sizeA

How many test cases are needed to trust that model A's accuracy really differs from model B's.

acc_a, acc_b: expected accuracies as proportions (e.g. 0.82, 0.78).
paired: True if both models run on the same test cases (usual case).
discordance: paired only -- fraction of cases where exactly one model is
  right. Measure it from a pilot run if you can; default assumes
  independent errors, which overestimates n for similar models.

Name	Required	Description	Default
`acc_a`	Yes
`acc_b`	Yes
`alpha`	No
`power`	No
`paired`	No
`discordance`	No

TDQS

A3.6/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the burden and does disclose important statistical behavior: discordance is paired-only, the default assumes independent errors, and this overestimates n for similar models. It still omits how alpha and power affect the calculation and what exactly is returned.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The purpose is front-loaded, followed by compact parameter notes; the pilot/discordance caveat is useful and earns its place. It is list-like but not bloated, with only minor missing structure around return values.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 6-parameter statistical calculator with no annotations or output schema, the description covers the key paired/discordance nuance but leaves alpha and power unexplained and does not explicitly state the return value. Adequate, but clear gaps remain.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It clearly defines acc_a/acc_b as proportions, paired as same test cases, and discordance as the fraction where exactly one model is right for paired designs, but alpha and power remain undocumented.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States that the tool computes the required test-case count to detect an accuracy difference between models A and B. The verb-equivalent and resource are clear, but it does not explicitly differentiate itself from sibling tools like eval_min_detectable_gap.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the planning question 'How many test cases are needed...' and by guidance that paired is the usual case. However, it never says when to choose this over eval_compare_paired, eval_compare_unpaired, or eval_min_detectable_gap.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

4 tool updatesv0.1.0Oct 23, 2026
- First observedeval_compare_paired
- First observedeval_compare_unpaired
- First observedeval_min_detectable_gap
- First observedeval_sample_size

Deploy Server

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

A3.6/5.0

Scored 2026-10-23 19:40 across 4 tools

Disambiguation5/5

Each tool has a clearly distinct statistical purpose: planning sample size, checking minimum detectable gap, paired comparison, and unpaired comparison. The paired vs. unpaired tests are explicitly distinguished by name and description, leaving little room for misselection.

Naming Consistency4/5

All tools share the eval_ prefix and snake_case formatting, which provides a predictable namespace. However, the suffixes mix verb-led names (eval_compare_paired) with noun phrases (eval_sample_size, eval_min_detectable_gap), a minor inconsistency.

Tool Count5/5

Four tools is well-scoped for a focused eval-statistics utility. Each tool covers a distinct and necessary part of sample-size planning and model comparison, with no redundant entries.

Completeness5/5

The surface covers the key lifecycle for eval statistical planning: determining required sample size, assessing detectable gaps, and comparing models under both paired and unpaired designs. No obvious gaps remain for the stated purpose.

Maintenance

ActivityMaintained

ResponsivenessNo issues

Resources

Related MCP Connectors

Hiring Fairness & Selection ROI AOAuth
ai.saifs
Detect hiring adverse impact and estimate whether a selection method is worth its cost.
Legal & Compliance Finance
21
https://hiring-fairness-mcp.saifs.ai/mcp
ML Referee B
com.mlreferee.api
Independent statistical feature evaluation for machine learning and forecasting. ML Referee tests up to 1,000 candidate features in one run against a declared baseline, helping researchers and AI agents decide which inputs to pursue, which bounded feature repairs merit further testing, and where more evidence is needed—without a separate ablation run for every feature. Use the MCP tools to discover input schemas, check access and credits, preflight a dataset, obtain a quote, submit an authorized evaluation, and retrieve structured results and reports. Results are scoped to the submitted evidence and evaluation design; they do not guarantee future model performance. Invite-only beta with manual approval. Public tool discovery requires no authentication; account and evaluation tools require OAuth through approved client connections and available evaluation credits. Connect directly to https://api.mlreferee.com/mcp. Website: https://mlreferee.com Request beta access: https://mlreferee.com/beta Agent setup and documentation: https://mlreferee.com/agents
AI & Machine Learning
261
https://api.mlreferee.com/mcp
databutler-stats A
io.github.physics-star-cat
Exact statistics & probability: distributions, hypothesis tests, CIs, Bayesian updates, regression.
Research & Data AI & Machine Learning
61
https://databutler.dev/api/mcp/stats
regimen A
io.github.RaYYeR220
Tells you whether a trading strategy's edge is distinguishable from luck.
Finance Research & Data
41
https://regimen-nu.vercel.app/mcp

Related MCP Servers

shewhart-mcp
Research & Data Data Visualization
bertanucar
A
license
Not graded
quality
D
maintenance
Enables AI agents to perform statistical process control calculations using validated, deterministic tools such as control charts, capability analysis, and tolerance intervals.
Updated 5 months ago (2026-06-12 11:25 UTC)
3
MIT
evalgateprivate
AI & Machine Learning Research & Data
ipezygj
A
license
A
quality
C
maintenance
Eval-integrity statistics for AI benchmark claims — multiple-testing correction, power/MDE for model gaps, judge-bias and leaderboard-rank checks. Catches a benchmark number that won't survive a second look.
Updated 3 months ago (2026-07-31 12:48 UTC)
9
MIT
agent-eval
Testing & QA Tools AI & Machine Learning
RudrenduPaul
A
license
A
quality
C
maintenance
MCP server exposing statistical regression testing for LLM agents as a "run" tool: p-value, effect size, and confidence interval on whether agent behavior actually changed.
Updated 7 days ago (2026-10-23 03:20 UTC)
1
62 PyPI
Apache 2.0
statistician-mcp
Research & Data Data Visualization
bmewing
A
license
Not graded
quality
C
maintenance
Provides a virtual statistician for AI agents, offering real statistical methods such as design of experiments, hypothesis testing, regression, and process control. It includes an advisor tool to recommend appropriate analyses and generates plain-language interpretations of results.
Updated 3 months ago (2026-08-13 12:57 UTC)
MIT

sample-size-mcp

CLI

MCP server

The statistics

Caveats

Available Tools

Tool Schema Changelog

Maintenance

Resources

Related MCP Connectors

Related MCP Servers

evalgateprivate