benchmark-ranking-stability

AI Benchmark Index · npm · SDK · HTTP API · MCP

benchmark-ranking-stability is an open-source model evaluation benchmark. It shows how rankings change across task samples, prompt wording, and repeated runs.

npx benchmark-ranking-stability run --provider mock --json

{"model":"mock:1","mean":0.5,"confidenceInterval95":{"low":0.44333333333333336,"high":0.5566666666666666},"totals":{"tasks":300,"passed":150,"failed":0},"breakdown":{"byFamily":{"multi-step-arithmetic":{"tasks":50,"mean":0.48},"ordering":{"tasks":50,"mean":0.52},"remainder":{"tasks":50,"mean":0.48},"rule-tracing":{"tasks":50,"mean":0.52},"sequence":{"tasks":50,"mean":0.52},"set-counting":{"tasks":50,"mean":0.48}},"byVariant":{"v1":{"tasks":60,"mean":0.5},"v2":{"tasks":60,"mean":0.5},"v3":{"tasks":60,"mean":0.5},"v4":{"tasks":60,"mean":0.5},"v5":{"tasks":60,"mean":0.5}}}}

The mock deliberately mixes correct and wrong answers. The 300 public tasks come from a committed seeded generator: 60 reasoning items with five paraphrases each.

Run a real model through OpenRouter:

OPENROUTER_API_KEY=your_key npx benchmark-ranking-stability run --provider openrouter --model openai/gpt-5-mini --json

Install

Node.js 20 or newer is required.

npm install benchmark-ranking-stability

Related MCP server: Kryve Agent Evaluation MCP

Measure ranking stability

Mode A is free and offline. Pass reports from at least two models:

npx benchmark-ranking-stability score-stability run-a.json run-b.json --seed 7 --svg volatility.svg

The command writes JSON statistics and a 1200×630 SVG. Each bootstrap resample draws items once and applies the draw to every model, then picks one run per model, so comparisons stay paired. A fixed seed makes it reproducible.

Mode B runs the included suite. For a real provider, run the same --model more than once to collect repeated reports; score-stability then resamples run-to-run variation as well as the task sample. The --seed flag drives only the bootstrap and the mock's answer pattern — providers never see it, and the mock encodes it in its model id (mock:N), so mock runs from different seeds compare as different models, not repeated runs.

How it works

Mode A: RunReport files -> align task IDs -> bootstrap tasks and runs
                                         -> rank intervals + tau-b + SVG

Mode B: 60 items × 5 prompts -> model -> normalized grader -> RunReport

The stability result reports:

• meanKendallTau: average agreement between the baseline and resampled rankings. One is stable; lower values mean more movement.

• probabilityFirst: the share of resamples where a model ranks first. Tied models split that resample.

• rankInterval95: the central 95% of bootstrapped ranks.

• mean: the model's mean task score across its supplied reports.

• promptSensitivity: per-variant rankings and each model's rank range, present when task IDs carry -vN variants.

• warnings: conditions that weaken the result, such as fewer than 30 shared tasks, one run per model, or under 500 resamples.

• sharedItems: the number of resampled items; paraphrase variants of one item count once.

Read intervals and first-place probabilities together. A small mean gap can coexist with uncertain rank order.

Why it exists

A leaderboard position can look exact when its underlying task and prompt choices are not. This package makes that sensitivity visible from ordinary benchmark reports.

The implementation builds on Kendall's treatment of rank ties and Efron's bootstrap. It does not claim a new statistical estimator.

Compare

Checked on 2026-10-15.

Coverage

The E2E gate runs all 300 tasks and every HTTP and MCP operation. It also checks weak policies: empty, constants, refusal, a repeated tool call, and a first-name heuristic must each score below 0.10, and answering with the first or last number in the prompt must stay below 0.10 in every family.

Common use cases

1. Evaluation teams: test whether a release winner survives task resampling.

2. Model teams: compare prompt variants before publishing a benchmark claim.

3. Researchers: attach rank intervals to repeated OpenRouter runs.

4. Reviewers: audit a ranking with fixed seeds and local report files.

5. Benchmark authors: find task samples that reverse close model rankings.

Features

• Two modes: analyze existing reports or run the included reasoning suite.

• Reproducible statistics: fixed seeds control 2,000 bootstrap resamples by default.

• Tie handling: Kendall tau-b and average ranks retain tied scores.

• Inspectable data: JSON Lines tasks and their generator ship in the package.

• Adversarial gate: empty, constant, refusal, repeated-tool-call, and first-name answers must score below 0.10.

• Shareable output: volatility and benchmark cards are plain SVG files.

SDK

import { benchmark, run, score, scoreStabilityFiles } from "benchmark-ranking-stability";

const result = await scoreStabilityFiles(
  ["run-a.json", "run-b.json"],
  { seed: 7, resamples: 2000, svgPath: "volatility.svg" },
);

benchmark exposes task metadata. run(adapter, options) runs Mode B and attaches a breakdown of means by family and variant; pass options.tasksPath to run a split file such as artifacts/hidden-tasks.jsonl. createBenchmark({ tasksPath }) binds the task list, score(taskId, submission) applies its deterministic grader, and reportBreakdown(report, tasks) recomputes the aggregates.

CLI

The standard commands are list, show, run, score, card, provenance, index-entry, serve, and mcp. This package also adds score-stability for Mode A.

Use run --provider mock without a key. Use --provider openrouter with OPENROUTER_API_KEY, or --provider openai with OPENAI_BASE_URL and OPENAI_API_KEY. Add --tasks <path> to list, show, run, score, serve, or mcp to work from a split file; run and score attach a breakdown of means by family and variant.

Model Context Protocol

Start the Model Context Protocol (MCP) server over standard input and output:

npx benchmark-ranking-stability mcp

Its tools are list_tasks, get_task, score_answer, run_benchmark, and get_provenance. run_benchmark returns the report with the same breakdown; pass provider: "mock" for a keyless offline run, and start the server as mcp --tasks <path> to serve a split file.

HTTP API

Import handler from benchmark-ranking-stability/api, or call createApi({ tasksPath }) for a handler bound to a split file. To self-host on Node:

npx benchmark-ranking-stability serve 3000

Routes are GET /health, GET /v1/tasks, GET /v1/tasks/:id, POST /v1/score, GET /v1/provenance, and GET /v1/index-entry.

Data provenance

scripts/gen-tasks.mjs creates all Mode B data under the MIT license. Seed 20261015 reproduces the committed public split.

node scripts/gen-tasks.mjs --seed 20261015 --split public
node scripts/gen-tasks.mjs --seed YOUR_PRIVATE_SEED --split hidden

The hidden command writes hid--prefixed tasks to ignored artifacts/hidden-tasks.jsonl and refuses to run without an explicit private --seed. Keep the file outside version control and run it with --tasks artifacts/hidden-tasks.jsonl. Refreshes happen quarterly after a reviewed public diff.

Mode A downloads no dataset. It reads only the RunReport files named on the command line. See data/PROVENANCE.md for the machine-aligned record.

Limits and known weaknesses

• The grader reads the first number token or an order-cued option mention as the answer; a stray value or name can be misread.

• Public tasks can enter training data. Use a privately seeded split for final claims. The hidden split stays hidden only while its seed is private and its prompts are not served publicly; anyone who learns the seed or sees the prompts can rebuild it.

• Five paraphrases of one item are correlated observations, so score-stability resamples items, not task rows.

• Small samples and ties can produce wide rank intervals or low-information rankings.

• Reports use the intersection of task IDs. Missing rows can shrink the effective sample.

• Different benchmark versions, graders, or provider settings can invalidate a comparison; 0.1.0 scores are not comparable to 0.2.0.

• The MCP score_answer tool declares submission as an open object schema in benchmark-core, so OpenAI strict tool mode may reject it.

• Bootstrap results describe the supplied reports. They do not cover every future prompt.

• The synthetic suite covers six compact reasoning families, not broad model quality.

The package has no public leaderboard yet. No real-model results are published because none were run for this release.

Integration guides

• Task grading and statistical assumptions

• Data source and refresh record

• TypeScript SDK

• Model Context Protocol

• HTTP API

Status

Version 0.2.0 is alpha. Schemas and statistical output can change before 1.0. Report defects through GitHub Issues.

Public case studies are not yet published.

FAQ

What does benchmark-ranking-stability measure?

benchmark-ranking-stability measures rank movement under resampled tasks, repeated runs, and prompt variants. It reports agreement, first-place probability, and rank intervals.

Can benchmark-ranking-stability analyze another benchmark?

benchmark-ranking-stability accepts RunReport JSON from any package in the benchmark family. Reports need model names, task IDs, and finite scores from zero to one.

Does benchmark-ranking-stability run offline?

benchmark-ranking-stability runs Mode A and the mock Mode B path without network access. Real model calls need an OpenRouter or OpenAI-compatible endpoint.

How many reports should I collect?

benchmark-ranking-stability accepts one report per model, but repeated runs let it resample run-to-run variation. More shared tasks and runs usually make the diagnostic more informative.

How does benchmark-ranking-stability handle ties?

benchmark-ranking-stability assigns average ranks to equal means. First-place ties split probability equally, and ranking agreement uses Kendall tau-b.

Does benchmark-ranking-stability provide a managed service?

benchmark-ranking-stability provides software you run. The HTTP server is self-hosted and has no built-in authentication.

Is the included suite a general intelligence test?

benchmark-ranking-stability uses a narrow synthetic suite to expose prompt sensitivity. It does not measure every capability or produce a universal model rank.

How is benchmark-ranking-stability different from lm-evaluation-harness?

benchmark-ranking-stability focuses on uncertainty in a ranking and can consume existing family reports. lm-evaluation-harness focuses on running a broad task catalog.

Why is there no public model leaderboard?

benchmark-ranking-stability has no verified real-model runs for version 0.2.0. The project does not invent or extrapolate scores.

License

MIT © Pooria Arab

Cite

Please cite this package with its Zenodo DOI. The DOI below names all versions. GitHub also shows a "Cite this repository" box from CITATION.cff.

@software{arab2026benchmarkrankingstability,
  author  = {Arab, Pooria},
  title   = {benchmark-ranking-stability},
  year    = {2026},
  version = {0.2.0},
  doi     = {10.5281/zenodo.23422385},
  url     = {https://aibenchmarkindex.com/benchmarks/benchmark-ranking-stability}
}