bench-registry

One process, one port, one SQLite file: an MCP server + REST API + zero-build web UI for logging training / benchmark / eval runs and querying leaderboards across machines.

Built for the pattern of "agent fleet runs benchmarks on several hosts, results scattered across logs" — every agent (or curl) appends a run with free-form metrics/config JSON, and anyone can ask "what was the best tokens/sec on any GPU for model X?" and get a ranked answer.

Reproduce in 3 commands

python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
.venv/bin/python server.py &          # UI on http://127.0.0.1:8100/ , MCP on /mcp
.venv/bin/python selftest.py          # raw JSON-RPC end-to-end test, no LLM needed

selftest.py prints PASS for initialize → tools/list → add → list → leaderboard → get → delete and cleans up after itself. Linux smoke-tested.

Related MCP server: metrillm-mcp

What you get

MCP tools

• bench_run_add(project, model, hardware, metrics, run_type, config, notes, status) — log a run; metrics/config are free-form JSON objects

• bench_run_list(project?, model?, hardware?, run_type?, limit?) — newest first

• bench_run_get(run_id) / bench_run_delete(run_id)

• bench_leaderboard(metric, project?, hardware?, model?, higher_is_better?, limit?) — rank runs by any numeric metric

• bench_summary() — counts per project/model/hardware + every numeric metric key that exists

Registering with an MCP client

Point any streamable-HTTP MCP client at http://127.0.0.1:8100/mcp. For Hermes:

hermes config set mcp_servers.bench-registry.url http://127.0.0.1:8100/mcp

(takes effect in new sessions)

Data model

Single runs table in runs.db (override with BENCH_REGISTRY_DB): id, created, project, run_type, model, hardware, status, config(json), metrics(json), notes. Port override: BENCH_REGISTRY_PORT (default 8100). No telemetry, no auth — keep it on localhost/LAN like any other local tool.

Why

Benchmark results across a fleet usually die in per-machine log files. This makes them a first-class queryable dataset that every agent on the network can write to and read from through its normal tool loop — and a human can eyeball in a browser.

License

MIT