benchmark-skateboard

AI Benchmark Index · npm · HTTP API · MCP

Can a model keep a kickflip, a heelflip, and gravity straight at the same time? benchmark-skateboard measures signed 3D trick rotations, projectile-motion arithmetic, wheels-down orientation, and whether a trick can finish before landing.

npx benchmark-skateboard run --provider mock

A real offline run on the 36 public tasks starts like this:

# Skateboard Physics

- Model: `mock:1`
- Score: **58.3%**
- 95% CI: 41.7%–72.2%
- Tasks: 36

The trick glossary is a human-written CC0 table checked against linked public trick definitions. A seeded MIT-licensed generator combines its 12 tricks with Earth, Moon, and Mars flight conditions.

Install

Node.js 20 or newer is required.

npm install benchmark-skateboard

Related MCP server: @mhdd_24/ai-benchmark-mcp

Run a model

The mock provider is deterministic, offline, and needs no key. It deliberately returns good and bad answers so you can inspect both grading paths.

For a real model through OpenRouter:

npx benchmark-skateboard run --provider openrouter --model openai/gpt-4o-mini

For another OpenAI-compatible endpoint, set OPENAI_BASE_URL and OPENAI_API_KEY, then choose --provider openai --model <id>. The package never reads keys from repository files.

There is no public leaderboard yet. The sample above is a wiring check, not a model result.

Read the score

Each task expects this JSON shape:

{"rotations":{"long":1,"vertical":0,"cross":0},"airtimeS":0.571,"peakHeightM":0.4,"landsWheelsDown":true,"canFlipFinishInAir":true}

The grader gives 36% to exact signed rotations, 34% to airtime and peak height within 5%, and 30% to exact landing and feasibility booleans. Wrong rotation signs lose credit: a heelflip is not a kickflip. Invalid JSON, missing fields, extra fields, and non-finite values score zero. Every row includes a reason, and the report includes a seeded bootstrap 95% confidence interval.

See grading for failure modes and weights and methodology for axes, equations, and assumptions.

Data and reproducibility

data/tricks.json contains the reviewed trick table and its source links. data/tasks.jsonl contains 36 public tasks: 12 tricks across three gravity and launch-speed variants. Rebuild them exactly:

node scripts/gen-tasks.mjs --seed 20261015 --split public

Generate a private candidate split with a different seed:

node scripts/gen-tasks.mjs --seed 43110 --split hidden

The hidden file is gitignored; keep it outside public reports and packages. data/PROVENANCE.md records licences, retrieval date, contamination cutoff, exact commands, and refresh cadence.

SDK

import { benchmark, run, score } from "benchmark-skateboard";

run(adapter, options) returns a standard RunReport. score(taskId, submission) applies the same deterministic grader used by every other surface.

CLI

The command set is list, show <task>, run, score <results.json>, card <report.json>, provenance, index-entry, serve, and mcp. Add --json for machine-readable output. card writes a 1200×630 SVG with the benchmark name, model, score, and directory URL.

Model Context Protocol

Start the stdio server:

npx benchmark-skateboard mcp

It exposes list_tasks, get_task, score_answer, run_benchmark, and get_provenance.

HTTP API

Import handler from benchmark-skateboard/api, or start a local server with npx benchmark-skateboard serve. The fetch-style handler works in Node.js and Cloudflare Workers.

Put authentication and rate limits in front of a public score endpoint.

Limits and known weaknesses

• The flight model ignores drag, deck inertia, foot contact, body rotation, and unequal landing height.

• Minimum trick times are declared benchmark assumptions, not measured athlete limits.

• Trick axes compress motions such as an impossible into one idealized rotation.

• Public prompts can enter training data. Use a fresh private seed for final comparisons.

• A 36-task sample has wide confidence intervals and should not support small ranking claims.

• The grader checks the stated model, not whether a generated animation looks convincing.

Prior art and sources

The coordinate framing was informed by the physical viewpoint in Skateboard Tricks and Topological Flips. Trick facts were checked against the pages named in data/tricks.json; equations follow OpenStax. Only short factual relationships were retained, in original wording.

Development

npm ci
npm run ci:local

The gate builds the package, rejects trivial answers, runs the mock over every task, calls every CLI and HTTP route, opens the MCP server with the official client, and writes artifacts/e2e-report.json.

Version 0.1.0 is alpha. Schemas and task assumptions may change before 1.0. Report defects in GitHub Issues.

License

MIT © Pooria Arab. The original glossary table is also offered under CC0-1.0 as recorded in data/tricks.json.