# evals-mcp-server

The primary sources (papers, project websites, official repositories) behind the AI benchmarks tracked by [BenchRisk](https://benchrisk.ai), indexed in Cloudflare AI Search and served to AI agents over the Model Context Protocol (MCP).

**Site:** https://averi-org.github.io/evals-mcp-server/ (benchmark table with filters, [Connect](https://averi-org.github.io/evals-mcp-server/connect.html), [About](https://averi-org.github.io/evals-mcp-server/about.html))

| Endpoint | URL |
| --- | --- |
| MCP (Streamable HTTP) | `https://ns-f1fc49e3-9bca-4aa2-b5a1-ef5dbc18588f.search.ai.cloudflare.com/mcp` |
| Search (ranked chunks) | `https://ns-f1fc49e3-9bca-4aa2-b5a1-ef5dbc18588f.search.ai.cloudflare.com/search` |
| Chat completions | `https://ns-f1fc49e3-9bca-4aa2-b5a1-ef5dbc18588f.search.ai.cloudflare.com/chat/completions` |

No API key is needed. The endpoints are rate limited to 1000 requests per 5 minutes.

## Quick start

```bash
# Claude Code
claude mcp add --transport http benchrisk-evals https://ns-f1fc49e3-9bca-4aa2-b5a1-ef5dbc18588f.search.ai.cloudflare.com/mcp

# Any MCP client that reads an mcpServers block
{"mcpServers": {"benchrisk-evals": {"type": "http", "url": "https://ns-f1fc49e3-9bca-4aa2-b5a1-ef5dbc18588f.search.ai.cloudflare.com/mcp"}}}

# Raw MCP call, restricted to one benchmark by its tag
curl -s https://ns-f1fc49e3-9bca-4aa2-b5a1-ef5dbc18588f.search.ai.cloudflare.com/mcp \
  -H "Content-Type: application/json" -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"search","arguments":{"query":"How is it scored?","ai_search_options":{"retrieval":{"filters":{"benchmark":"mmlu"}}}}}}'
```

The server exposes one tool, `search`. Every indexed chunk carries these metadata fields, all usable in `ai_search_options.retrieval.filters`:

| Field | Meaning | Example |
| --- | --- | --- |
| `benchmark` | Unique tag for the benchmark | `mmlu`, `ailuminate-1.0`, `big-bench-hard` |
| `benchmark_name` | The benchmark's name in BenchRisk | `MMLU`, `AILuminate10`, `BigBenchHard` |
| `doc_type` | `paper`, `website`, or `repository` | `paper` |
| `source_url` | Where the document came from | `https://arxiv.org/abs/2009.03300` |
| `benchrisk_url` | The BenchRisk entry the document was derived from | GitHub URL of the `.mdx` score file |
| `folder` (built in) | Always `<tag>/` | `mmlu/` |

The full list of tags is in [`benchmarks.json`](benchmarks.json) and on the site.

## What is indexed

Only primary sources. For each benchmark BenchRisk scores (`data/scores/*.mdx` in [AVERI-org/BenchRisk](https://github.com/AVERI-org/BenchRisk)):

1. **The definitive reference** BenchRisk cites. For arXiv references this is the PDF, or the arXiv HTML rendering when the PDF exceeds Cloudflare's 4 MiB limit. For project websites it is the page HTML.
2. **The official repository** when the BenchRisk entry names one (README via the GitHub API).

Nothing generated from BenchRisk itself is indexed; BenchRisk supplies the list, descriptions, references and tags, which live in `benchmarks.json` and on the site. Items are stored under the key `<tag>/<file>` so the built-in `folder` attribute doubles as the tag. Benchmarks that BenchRisk marks `hide: true` are still indexed and flagged on the site.

## Repository layout

```
benchmarks.json                 manifest: every benchmark, tag, reference, and indexed item key
config/cloudflare.json          namespace, instance, public endpoints, metadata schema, MCP tool description
config/benchmarks.overrides.json  BenchRisk name -> tag, reference fallbacks, extra resources
scripts/build_manifest.py       BenchRisk repo -> benchmarks.json
scripts/fetch_resources.py      benchmarks.json -> .cache/resources/<tag>/<file> (+ keys back into the manifest)
scripts/upload_to_cloudflare.py .cache -> Cloudflare AI Search instance (uploads changes, prunes items not in the manifest), then expose it on the namespace endpoint
scripts/verify_seed.py          one search per benchmark through the public MCP endpoint
scripts/build_site.py           benchmarks.json + site/ -> _site/ (table, connect, about, one page per benchmark)
site/style.css                  the site stylesheet
tests/                          pytest suite run by the Seed accessibility workflow
.github/workflows/              seed-accessibility.yml, pages.yml, sync-cloudflare.yml
```

## Running the pipeline locally

```bash
python3 -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt

export CLOUDFLARE_API_TOKEN=...      # AI Search:Edit + AI Search:Run
export CLOUDFLARE_ACCOUNT_ID=...
export BENCHRISK_GITHUB_TOKEN=...    # optional, raises GitHub rate limits

python scripts/build_manifest.py          # read BenchRisk, assign tags, write benchmarks.json
python scripts/fetch_resources.py         # download papers, sites and READMEs
python scripts/upload_to_cloudflare.py    # idempotent upload + prune + wait for indexing + allowlist
python scripts/verify_seed.py             # quick public check
pytest -v                                 # full suite (authenticated tests skip without the env vars)
python scripts/build_site.py && python -m http.server -d _site 8000
```

`upload_to_cloudflare.py` records the sha256 of every uploaded file in `benchmarks.json` (`uploaded_sha256`) and only re-uploads what changed or failed. It also deletes any item in the instance that is not in the manifest, so the instance always holds exactly the manifest's primary sources (`--no-prune` to keep extras). Use `--force` to re-upload everything. `pdftotext` (poppler-utils) is only needed if an arXiv paper has neither a PDF under 4 MiB nor an HTML rendering.

### Adding or renaming a benchmark

When BenchRisk adds a score file, `build_manifest.py` stops until the new name has a tag in `config/benchmarks.overrides.json`. Tags are permanent identifiers: lowercase, digits, hyphens and dots. If a reference site is unreachable, add a `reference_fallbacks` entry with the reason; it is shown on the site and in the benchmark card.

## GitHub Actions

| Workflow | Trigger | What it does |
| --- | --- | --- |
| **Seed accessibility** | push, pull request, daily, manual | Runs `tests/`: MCP initialize and tool listing, a filtered search per benchmark over MCP and `/search`, a `/chat/completions` smoke test, and, with the Cloudflare secrets, checks that every manifest item finished indexing with the right metadata, that the instance holds nothing else, and that it is on the namespace allowlist. |
| **Deploy GitHub Pages** | push to `main` touching the site or manifest, manual | Renders `_site/` and deploys it. |
| **Sync Cloudflare index from BenchRisk** | manual | Rebuilds the manifest from BenchRisk, fetches resources, uploads new or changed items, removes items no longer listed, verifies, commits the refreshed manifest, and redeploys the site. |

### Repository secrets

| Secret | Used by |
| --- | --- |
| `CLOUDFLARE_API_TOKEN` | upload, sync, authenticated tests |
| `CLOUDFLARE_ACCOUNT_ID` | upload, sync, authenticated tests |
| `BENCHRISK_GITHUB_TOKEN` | sync (reading BenchRisk; the repo is public, so this only raises rate limits) |

## Cloudflare layout

- Namespace `benchrisk`, public endpoint id `ns-f1fc49e3-9bca-4aa2-b5a1-ef5dbc18588f`, with `instances_allowed = ["benchmarks"]`.
- Instance `benchmarks`: built-in storage, hybrid (vector + BM25) retrieval, `@cf/qwen/qwen3-embedding-0.6b` embeddings, five custom metadata fields (`benchmark`, `benchmark_name`, `doc_type`, `source_url`, `benchrisk_url`).
- Management uses the [AI Search REST API](https://developers.cloudflare.com/ai-search/api/instances/rest-api/) under `/accounts/{account_id}/ai-search/namespaces/benchrisk/instances/benchmarks/`.

## License

Code in this repository is MIT licensed (see `LICENSE`). Indexed papers and web pages remain the property of their authors and are indexed for retrieval only.