benchmark-skateboard

AI Benchmark Index · npm · HTTP API · MCP

Can a model keep a kickflip, a heelflip, and gravity straight at the same time? benchmark-skateboard measures signed 3D trick rotations, projectile-motion arithmetic, wheels-down orientation, and whether a trick can finish before landing.

npx benchmark-skateboard run --provider mock

A real offline run on the 72 public tasks starts like this:

# Skateboard Physics

- Model: `mock:1`
- Score: **54.2%**
- 95% CI: 41.7%–65.3%
- Tasks: 72

Breakdown — rotations: 13.0%, physics: 20.6%, landing: 20.6%

The trick glossary is a human-written CC0 table checked against linked public trick definitions. A seeded MIT-licensed generator combines its 24 tricks with Earth, Moon, and Mars flight conditions.

Install

Node.js 20 or newer is required.

npm install benchmark-skateboard

Related MCP server: benchmark-test-weakening

Run a model

The mock provider is deterministic, offline, and needs no key. It deliberately returns good and bad answers so you can inspect both grading paths.

For a real model through OpenRouter:

npx benchmark-skateboard run --provider openrouter --model openai/gpt-4o-mini

For another OpenAI-compatible endpoint, set OPENAI_BASE_URL and OPENAI_API_KEY, then choose --provider openai --model <id>. The package never reads keys from repository files.

There is no public leaderboard yet. The sample above is a wiring check, not a model result.

Read the score

Each task expects this JSON shape:

{"rotations":{"long":1,"vertical":0,"cross":0},"airtimeS":0.571,"peakHeightM":0.4,"landsWheelsDown":true,"canFlipFinishInAir":true}

The grader gives 24% to exact signed rotations together, 38% to airtime and peak height within 5%, and 38% to the landing and feasibility booleans once both physics values are within tolerance. landsWheelsDown is scored against the expected rotation geometry and canFlipFinishInAir against the expected airtime, so half-flip tricks and underpowered launches cannot be answered from memory. One surrounding Markdown fence or prose around the JSON is fine; when several objects appear, the last balanced one wins. Wrong rotation signs lose the whole rotation block. Invalid JSON, missing fields, extra fields, and non-finite values score zero. Every row includes a reason; run and score also print rotations/physics/landing breakdown means plus a bootstrap 95% confidence interval.

See grading for failure modes and weights and methodology for axes, equations, and assumptions.

Data and reproducibility

data/tricks.json contains the reviewed trick table and its source links. data/tasks.jsonl contains 72 public tasks: 24 tricks across three gravity and launch-speed variants, including half flips that cannot land wheels-down and launches too weak to finish the rotation. Rebuild them exactly:

node scripts/gen-tasks.mjs --seed 20261015 --split public

Generate a private candidate split with a different seed. The hidden split requires an explicit --seed and refuses the public seed 20261015:

node scripts/gen-tasks.mjs --seed 88200416 --split hidden

The hidden file lands in .hidden/tasks.jsonl (gitignored) and is excluded from the npm package. Point a run at another split with --tasks <path> on the CLI (including serve and mcp), run(adapter, { tasks }) or createBenchmark(path) in the SDK, ?tasks=<path> on each HTTP route, or a tasksPath argument on MCP tools. Remote surfaces (?tasks= and tasksPath) only accept *.jsonl files under data/, so a hidden split stays reachable only through the CLI flag and the SDK option. data/PROVENANCE.md records licences, retrieval date, contamination cutoff, exact commands, and refresh cadence.

SDK

import { benchmark, run, score } from "benchmark-skateboard";

run(adapter, options) returns a standard RunReport plus the skateboard field breakdown; options.tasks points it at a split file. createBenchmark(path) binds a new instance to one tasks file — there is no process-global tasks path. score(taskId, submission) applies the same deterministic grader used by every other surface.

CLI

The command set is list, show <task>, run, score <results.json>, card <report.json>, provenance, index-entry, serve, and mcp. Add --json for machine-readable output. card writes a 1200×630 SVG with the benchmark name, model, score, and directory URL.

Model Context Protocol

Start the stdio server:

npx benchmark-skateboard mcp

It exposes list_tasks, get_task, score_answer, run_benchmark, and get_provenance. run_benchmark accepts provider (openrouter, openai, or mock) and returns the report with the rotations/physics/landing breakdown. Every task-reading tool accepts an optional tasksPath naming a *.jsonl split file under data/.

HTTP API

Import handler from benchmark-skateboard/api, or start a local server with npx benchmark-skateboard serve. The fetch-style handler works in Node.js and Cloudflare Workers.

Add ?tasks=<path> to any route to read a public split file under data/ instead of the default tasks.

Put authentication and rate limits in front of a public score endpoint.

Limits and known weaknesses

• The flight model ignores drag, deck inertia, foot contact, body rotation, and unequal landing height.

• Minimum trick times are declared benchmark assumptions, not measured athlete limits.

• Trick axes compress motions such as an impossible into one idealized rotation.

• Public prompts can enter training data. Use a fresh private seed for final comparisons.

• A 72-task sample has wide confidence intervals and should not support small ranking claims.

• The grader checks the stated model, not whether a generated animation looks convincing.

Prior art and sources

The coordinate framing was informed by the physical viewpoint in Skateboard Tricks and Topological Flips. Trick facts were checked against the pages named in data/tricks.json; equations follow OpenStax. Only short factual relationships were retained, in original wording.

Development

npm ci
npm run ci:local

The gate builds the package, rejects trivial answers, runs the mock over every task, calls every CLI and HTTP route, opens the MCP server with the official client, and writes artifacts/e2e-report.json.

Version 0.2.0 is alpha. Schemas and task assumptions may change before 1.0. Report defects in GitHub Issues.

License

MIT © Pooria Arab. The original glossary table is also offered under CC0-1.0 as recorded in data/tricks.json.

Cite

Please cite this package with its Zenodo DOI. The DOI below names all versions. GitHub also shows a "Cite this repository" box from CITATION.cff.

@software{arab2026benchmarkskateboard,
  author  = {Arab, Pooria},
  title   = {benchmark-skateboard},
  year    = {2026},
  version = {0.2.0},
  doi     = {10.5281/zenodo.23422365},
  url     = {https://aibenchmarkindex.com/benchmarks/benchmark-skateboard}
}