Time-Travel Benchmark

You can freeze the internet at any past date and let an LLM do real research inside that frozen world. This repo is a working rig for exactly that: a time-lock proxy over OpenReward's backdated search (built by General Reasoning), sealed model runners, and a scoring harness, demonstrated by locking five frontier models inside June 10, 2026 and making them predict the 2026 FIFA World Cup one day before kickoff.

Why bother? Because evaluating a model's judgment about the future is usually impossible: by the time you know the answer, every model has read it. Backdated search removes the lookahead. The model searches, fetches, and cites a web that ends on a date you choose, so you can backtest forecasting agents against any past event, benchmark reasoning with zero contamination from the outcome, or reconstruct exactly what the world knew the day before anything happened. The World Cup run here is the worked example, not the product.

Reality, for the record: Spain beat Argentina 1-0 in the final on July 19, 2026. Three of the five sealed models called Spain.

The sealed leaderboard, graded against the real result. Every model was locked to the same June 10 snapshot with the same tool budget.

The tricky parts

The cutoff must be unforgeable

Giving a model a "please pretend it's June 10" prompt proves nothing; it has to be physically unable to see past the date. The proxy in src/mcp-server.mjs exposes exactly two tools, web_search and web_fetch, and injects as_of: 2026-06-10 server side on every OpenReward call. The parameter never appears in the tool schema, so the model cannot change it or even see it. On top of OpenReward's own enforcement, the proxy re-checks every returned crawl timestamp and withholds anything that postdates the cutoff (in the five runs, that second filter caught 0 pages, which is what you want to see). Budgets are enforced in the same place: 10 searches, 15 fetches, 25 calls total per model, with the remaining budget echoed after each call so the model can plan.

Sealing a leaky harness

Four of the models ran through the Claude Code CLI harness (the Claude pair directly, MiniMax and Qwen via Anthropic-compatible endpoints), and that harness ships with tools that would break the seal instantly: live web search, shell, file system. src/run-model.mjs disallows every built-in tool and passes a strict MCP config so the time-lock proxy is the only tool surface left. GPT was the awkward one. The plan was Codex CLI in its read-only sandbox, and the proxy's streamable HTTP mode on localhost (WC_HTTP_PORT) exists for that, but codex exec auto-cancels custom MCP tool calls. So the shipped GPT run uses src/run-gpt-api.mjs instead: a direct OpenAI Responses API loop where the only tools the model receives are web_search and web_fetch as function definitions, wired to the same localhost proxy, so the caps, the cutoff, and the event log all stay server side. Each model also gets its own proxy instance, so budgets and event logs never mix.

Models invent citations, so citations are verified

Every tool call is appended to a JSONL event log: queries, returned URLs, titles, crawl dates. After a run, src/schema.mjs validates the model's prediction against that log. A citation only counts if its URL actually appeared in that model's own tool results and its crawl date is on or before the cutoff. Same for every sourceUrl behind every decisive signal. Hallucinated references fail validation instead of decorating the writeup. Across the five runs the models made 31 archived searches and 37 archived fetches, and all 41 citations in the final artifact trace back to logged tool results.

Reality is quarantined from the sealed phase

The true outcome lives in one module, src/actual-result.mjs, and exactly one thing imports it: the scoring step, which runs after all sealed runs complete. The proxy, the runners, and the orchestrator have no path to it, and test/seal.test.mjs enforces that: it fails the suite if any sealed module ever references the result module, or if the prompt leaks the outcome.

The caveat you cannot engineer away

Web access can be frozen. Weights cannot. A model released after the tournament may have absorbed the result during training, and no prompt guarantees it ignores that. Every contestant therefore carries a badge based on its public release date versus the cutoff:

Treat the "web-locked only" rows as an exhibit, not as evidence of foresight. The pre-event models are the canonical experiment. There is one encouraging detail: GPT-5.6 Sol was released three weeks after the final and still picked France, which is the behavior you'd expect if the lock, not the weights, was driving the research.

Related MCP server: smart-webfetch-mcp

Results

Scored with the Futuresight Score (0 to 100): 50 points for the correct champion, 20 for the correct runner-up, 7.5 per correct semifinalist compared as a set. Ties break on champion confidence, then semifinalist count. Dark horse and disappointment picks were for entertainment only.

Qwen3.8 Max named the exact final, Spain over Argentina. Opus 4.8 and Fable 5 both nailed all four real semifinalists. Every model got at least three of the four.

The winning run. Qwen3.8 Max called Spain over Argentina, the exact final, from inside the frozen snapshot.

The control that matters: GPT-5.6 Sol was released after the tournament ended and still predicted France, suggesting the sealed web, not memorized results, shaped the answers.

Repository layout

Running it

npm install
cp .env.example .env               # then fill in your keys
node src/run-experiment.mjs        # all sealed runs (needs the model CLIs installed)
node src/score.mjs                 # scores a fresh run, or the shipped data/ artifact
node src/build-cards.mjs           # renders the cards (npx playwright install chromium first)
npm test                           # scoring tests + the seal test

Scoring and card rendering work straight from a clone: both fall back to the sanitized artifacts in data/ when no fresh runs/ output exists. A default experiment costs at most 25 OpenReward calls per model (125 for the full five), minus cache hits. The runners assume locally installed CLI harnesses plus local endpoint and key files for MiniMax and Qwen; swap in your own model access in src/models.mjs and src/run-model.mjs.

Point it at your own event

The rig is not football specific; the card renderer is. To replay a different event, change the cutoff date (it lives in src/mcp-server.mjs, src/models.mjs, src/schema.mjs, and src/prompt.md, all four), rewrite the prompt, and set the outcome in src/actual-result.mjs. The same machinery then covers any past event: an election, an earnings call, an award season, a product launch. Expect to rewrite src/build-cards.mjs too, since its layout and copy are baked for this tournament. The expensive part, a searchable web frozen at an arbitrary date, is OpenReward's problem, and they solved it.

Credits

Backdated search and fetch by OpenReward, built by General Reasoning. The whole experiment leans on their as_of parameter doing what it says.

License

MIT. See LICENSE.