Skip to main content
Glama
fayehall-ai

eval-sample-size

by fayehall-ai
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Remote

Server Configuration

Describes the environment variables required to run the server.

NameRequiredDescriptionDefault

No arguments

Instructions

Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.

This server publishes no instructions, or was last inspected before Glama recorded them.

Capabilities

Features and capabilities supported by this server

Protocol revision2025-11-25

CapabilityDetails
tools
{
  "listChanged": false
}
prompts
{
  "listChanged": false
}
resources
{
  "subscribe": false,
  "listChanged": false
}

Tools

Functions exposed to the LLM to take actions

NameDescription
eval_sample_sizeA

How many test cases are needed to trust that model A's accuracy really differs from model B's.

acc_a, acc_b: expected accuracies as proportions (e.g. 0.82, 0.78).
paired: True if both models run on the same test cases (usual case).
discordance: paired only -- fraction of cases where exactly one model is
  right. Measure it from a pilot run if you can; default assumes
  independent errors, which overestimates n for similar models.
eval_min_detectable_gapC

Smallest accuracy improvement over baseline_acc that an eval with n test cases can reliably detect. Use this to sanity-check an existing eval set.

eval_compare_pairedA

Is A's win over B on a shared test set statistically real?

n: total cases; a_only: A right & B wrong; b_only: B right & A wrong;
both: both right. Returns exact McNemar p-value and a CI on the gap.
eval_compare_unpairedC

Is A's win over B real when each was scored on a different sample? Two-proportion z-test with a CI on the gap.

Prompts

Interactive templates invoked by user choice

NameDescription

No prompts

Resources

Contextual data attached and managed by the client

NameDescription

No resources

Deploy Server
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

A3.6/5.0

Scored across 4 tools

Disambiguation5/5

Each tool has a clearly distinct statistical purpose: planning sample size, checking minimum detectable gap, paired comparison, and unpaired comparison. The paired vs. unpaired tests are explicitly distinguished by name and description, leaving little room for misselection.

Naming Consistency4/5

All tools share the eval_ prefix and snake_case formatting, which provides a predictable namespace. However, the suffixes mix verb-led names (eval_compare_paired) with noun phrases (eval_sample_size, eval_min_detectable_gap), a minor inconsistency.

Tool Count5/5

Four tools is well-scoped for a focused eval-statistics utility. Each tool covers a distinct and necessary part of sample-size planning and model comparison, with no redundant entries.

Completeness5/5

The surface covers the key lifecycle for eval statistical planning: determining required sample size, assessing detectable gaps, and comparing models under both paired and unpaired designs. No obvious gaps remain for the stated purpose.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Resources