evals-mcp-server

The primary sources (papers, project websites, official repositories) behind the AI benchmarks tracked by BenchRisk, indexed in Cloudflare AI Search and served to AI agents over the Model Context Protocol (MCP).

Site: https://averi-org.github.io/evals-mcp-server/ (benchmark table with filters, Connect, About)

No API key is needed. The endpoints are rate limited to 1000 requests per 5 minutes.

Quick start

# Claude Code
claude mcp add --transport http benchrisk-evals https://ns-f1fc49e3-9bca-4aa2-b5a1-ef5dbc18588f.search.ai.cloudflare.com/mcp

# Any MCP client that reads an mcpServers block
{"mcpServers": {"benchrisk-evals": {"type": "http", "url": "https://ns-f1fc49e3-9bca-4aa2-b5a1-ef5dbc18588f.search.ai.cloudflare.com/mcp"}}}

# Raw MCP call, restricted to one benchmark by its tag
curl -s https://ns-f1fc49e3-9bca-4aa2-b5a1-ef5dbc18588f.search.ai.cloudflare.com/mcp \
  -H "Content-Type: application/json" -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"search","arguments":{"query":"How is it scored?","ai_search_options":{"retrieval":{"filters":{"benchmark":"mmlu"}}}}}}'

The server exposes one tool, search. Every indexed chunk carries these metadata fields, all usable in ai_search_options.retrieval.filters:

The full list of tags is in benchmarks.json and on the site.

Related MCP server: aether-developer

What is indexed

Only primary sources. For each benchmark BenchRisk scores (data/scores/*.mdx in AVERI-org/BenchRisk):

1. The definitive reference BenchRisk cites. For arXiv references this is the PDF, or the arXiv HTML rendering when the PDF exceeds Cloudflare's 4 MiB limit. For project websites it is the page HTML.

2. The official repository when the BenchRisk entry names one (README via the GitHub API).

Nothing generated from BenchRisk itself is indexed; BenchRisk supplies the list, descriptions, references and tags, which live in benchmarks.json and on the site. Items are stored under the key <tag>/<file> so the built-in folder attribute doubles as the tag. Benchmarks that BenchRisk marks hide: true are still indexed and flagged on the site.

Repository layout

benchmarks.json                 manifest: every benchmark, tag, reference, and indexed item key
config/cloudflare.json          namespace, instance, public endpoints, metadata schema, MCP tool description
config/benchmarks.overrides.json  BenchRisk name -> tag, reference fallbacks, extra resources
scripts/build_manifest.py       BenchRisk repo -> benchmarks.json
scripts/fetch_resources.py      benchmarks.json -> .cache/resources/<tag>/<file> (+ keys back into the manifest)
scripts/upload_to_cloudflare.py .cache -> Cloudflare AI Search instance (uploads changes, prunes items not in the manifest), then expose it on the namespace endpoint
scripts/verify_seed.py          one search per benchmark through the public MCP endpoint
scripts/build_site.py           benchmarks.json + site/ -> _site/ (table, connect, about, one page per benchmark)
site/style.css                  the site stylesheet
tests/                          pytest suite run by the Seed accessibility workflow
.github/workflows/              seed-accessibility.yml, pages.yml, sync-cloudflare.yml

Running the pipeline locally

python3 -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt

export CLOUDFLARE_API_TOKEN=...      # AI Search:Edit + AI Search:Run
export CLOUDFLARE_ACCOUNT_ID=...
export BENCHRISK_GITHUB_TOKEN=...    # optional, raises GitHub rate limits

python scripts/build_manifest.py          # read BenchRisk, assign tags, write benchmarks.json
python scripts/fetch_resources.py         # download papers, sites and READMEs
python scripts/upload_to_cloudflare.py    # idempotent upload + prune + wait for indexing + allowlist
python scripts/verify_seed.py             # quick public check
pytest -v                                 # full suite (authenticated tests skip without the env vars)
python scripts/build_site.py && python -m http.server -d _site 8000

upload_to_cloudflare.py records the sha256 of every uploaded file in benchmarks.json (uploaded_sha256) and only re-uploads what changed or failed. It also deletes any item in the instance that is not in the manifest, so the instance always holds exactly the manifest's primary sources (--no-prune to keep extras). Use --force to re-upload everything. pdftotext (poppler-utils) is only needed if an arXiv paper has neither a PDF under 4 MiB nor an HTML rendering.

Adding or renaming a benchmark

When BenchRisk adds a score file, build_manifest.py stops until the new name has a tag in config/benchmarks.overrides.json. Tags are permanent identifiers: lowercase, digits, hyphens and dots. If a reference site is unreachable, add a reference_fallbacks entry with the reason; it is shown on the site and in the benchmark card.

GitHub Actions

Repository secrets

Cloudflare layout

• Namespace benchrisk, public endpoint id ns-f1fc49e3-9bca-4aa2-b5a1-ef5dbc18588f, with instances_allowed = ["benchmarks"].

• Instance benchmarks: built-in storage, hybrid (vector + BM25) retrieval, @cf/qwen/qwen3-embedding-0.6b embeddings, five custom metadata fields (benchmark, benchmark_name, doc_type, source_url, benchrisk_url).

• Management uses the AI Search REST API under /accounts/{account_id}/ai-search/namespaces/benchrisk/instances/benchmarks/.

License

Code in this repository is MIT licensed (see LICENSE). Indexed papers and web pages remain the property of their authors and are indexed for retrieval only.