Skip to main content
Glama
fayehall-ai

eval-sample-size

by fayehall-ai
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Remote

eval_sample_size

Calculate how many test cases are needed to trust that one model’s accuracy differs from another’s, including paired or unpaired comparisons and desired significance and power.

Instructions

How many test cases are needed to trust that model A's accuracy really differs from model B's.

acc_a, acc_b: expected accuracies as proportions (e.g. 0.82, 0.78).
paired: True if both models run on the same test cases (usual case).
discordance: paired only -- fraction of cases where exactly one model is
  right. Measure it from a pilot run if you can; default assumes
  independent errors, which overestimates n for similar models.

Input Schema

NameRequiredDescriptionDefault
acc_aYes
acc_bYes
alphaNo
powerNo
pairedNo
discordanceNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.1.0

TDQS

A3.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the burden and does disclose important statistical behavior: discordance is paired-only, the default assumes independent errors, and this overestimates n for similar models. It still omits how alpha and power affect the calculation and what exactly is returned.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The purpose is front-loaded, followed by compact parameter notes; the pilot/discordance caveat is useful and earns its place. It is list-like but not bloated, with only minor missing structure around return values.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 6-parameter statistical calculator with no annotations or output schema, the description covers the key paired/discordance nuance but leaves alpha and power unexplained and does not explicitly state the return value. Adequate, but clear gaps remain.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It clearly defines acc_a/acc_b as proportions, paired as same test cases, and discordance as the fraction where exactly one model is right for paired designs, but alpha and power remain undocumented.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States that the tool computes the required test-case count to detect an accuracy difference between models A and B. The verb-equivalent and resource are clear, but it does not explicitly differentiate itself from sibling tools like eval_min_detectable_gap.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the planning question 'How many test cases are needed...' and by guidance that paired is the usual case. However, it never says when to choose this over eval_compare_paired, eval_compare_unpaired, or eval_min_detectable_gap.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools