Skip to main content
Glama

agentic_multiverse_analysis

Agentic multiverse analysis of an empirical paper, as an MCP server (ama-mcp) and a command line (ama).

ama-mcp runs an agentic multiverse analysis (AMA) of one published empirical paper and reports how much of the paper's defensible specification space keeps its main conclusion. It is the pipeline used for the two accompanying papers (see Citation), packaged as an MCP server (ama-mcp) and a command line (ama). Every role in the pipeline runs as a fresh agent session through your own Claude Code (claude) or OpenAI Codex (codex) CLI login.

Safety first. The agent sessions run code from replication packages, install packages, download data and browse the web, with permission prompts turned off. Run ama-mcp inside a container or a virtual machine that holds only what the analysis needs (the paper, the data you are licensed to use, the agent CLI login). Do not run it on a machine whose files or credentials you would not hand to an unattended agent. See Safety.

What AMA does

You give it a paper. It then:

  1. Finds the authors' replication package (journal archives, data repositories, author pages). If there is no complete package, an agent rebuilds the data from the original sources you can access, following the paper's own description, and an independent reviewer checks every sample rule and reported sample size against the paper.

  2. Pins down the main estimate. One agent writes a configuration naming the paper's headline coefficient, its table location and the claimed sign; another agent checks it against the paper.

  3. Reproduces the main estimate with its own code. A gate script compares the result with the published number (within 10% by default). On a miss, an independent auditor looks for implementation errors before the run may continue.

  4. Lists the analysis choices. One agent lists every choice the analysis makes (sample, variable definitions, controls, fixed effects, estimator, inference, ...) and the defensible alternatives. A different agent judges each option against a fixed rubric and designs a grid of 100 to 250 specifications, and fixes the inference method, before any alternative estimate exists.

  5. Runs the grid. An implementer writes and runs the grid; an independent auditor reads the code line by line and recomputes cells; fix rounds continue until the auditor finds no blocking problem.

  6. Measures what the auditor's exclusions cost the paper (a "companion" grid with the paper's own choices restored).

  7. Reports. The papers' measures are computed by scripts; an agent collects what the authors say about the decisive choices, and a reporter writes a report whose every number is regenerated by a script and checked byte for byte.

Three rules make the output worth reading: no agent checks its own work; the space of choices is judged before any estimate exists; every step ends at a gate decided by a program or by an independent agent, never by the agent whose work is gated. Gate decisions are appended to gates.log, and artifacts are hashed when a gate passes so a later edit is detected.

The prompts, gate scripts and orchestration contract in src/ama_mcp/core/ are the ones the study ran, changed only where a public user's machine needs it. Every change is listed, with its reason, in docs/CHANGES_FROM_STUDY.md.

Related MCP server: ai-architect

Requirements

  • Linux or macOS, Python 3.10 or newer.

  • At least one agent CLI, installed and logged in:

    • Claude Code: claude on your PATH; log in once interactively (claude, then /login) or set an API key as Claude Code documents.

    • OpenAI Codex CLI: codex on your PATH; codex login.

  • A Python environment for the agents' analysis code. The agents write Python by default; a typical environment has pandas, numpy, statsmodels, linearmodels, pyfixest, wildboottest, pyreadstat, pyarrow and matplotlib. Pass it with --analysis-python. Stata or R are used only if a replication package needs them and they are installed.

  • Access to the data the paper uses, unless the authors' package ships it (see Data access).

Install

python -m venv .venv
.venv/bin/pip install .            # from a clone of this repository
.venv/bin/ama doctor               # checks the core files, Python packages and agent CLIs

ama doctor reports whether each CLI is installed and logged in, whether the runs directory is writable, and whether it is running inside a container.

Use it from Claude Code or Codex (MCP)

Register the server once. Use the absolute path of the ama-mcp executable in your environment if it is not on your PATH (ama-mcp --help prints both registrations with that path filled in).

Claude Code

claude mcp add --scope user ama -e AMA_RUNS_ROOT="$HOME/ama-runs" -- /path/to/.venv/bin/ama-mcp

Codex (~/.codex/config.toml)

[mcp_servers.ama]
command = "/path/to/.venv/bin/ama-mcp"
env = { AMA_RUNS_ROOT = "/path/to/ama-runs" }

Then ask your agent, for example: "Use the ama tools to run an agentic multiverse analysis of ~/papers/paper.pdf with the claude backend." The server also ships an MCP prompt, run_analysis, that walks an agent through the steps.

The backend that runs the analysis is independent of the client you talk to: a Codex session can start a run on the claude backend and the other way round.

Tool

What it does

check_backends

Is claude / codex installed and logged in? (never returns credentials)

analyze_paper

Creates a run directory and starts the analysis in the background; returns a run_id. Takes the options of ama run, including effort

run_status

Status, current stage, gate outcomes so far, the headline once it exists, the last log lines

run_results

ama_result.json, the headline (share supported first), the strict survival rate, all measures, a report excerpt

list_runs

All runs, newest first

stop_run / resume_run

Stop a run; continue it from where it stopped

read_run_file

Read a text file of a run (the held-out benchmark/ folder is never readable)

Resources: the pipeline documents (ama://docs/...) and the 18 role prompt templates (ama://prompts/...).

Use it from the command line

ama run paper.pdf --backend claude \
    --citation "Author and Author (2015), Title, Journal" \
    --data-sources my_data_sources.md --analysis-python /path/to/analysis-env/bin/python
ama status                 # all runs
ama status <run_id>        # one run: stage, gates, headline, last log lines
ama results <run_id>       # headline, strict survival, measures (--all, --definitions), report excerpt
ama stop <run_id>
ama resume <run_id>        # continue after a stop, an error or a pause
ama doctor

ama run starts the analysis in the background and returns; add --foreground to keep it attached to the terminal. Runs live in $AMA_RUNS_ROOT (default ~/ama-runs). The orchestrator can also be driven directly: python -m ama_mcp.orchestrator run --run-dir RUN.

Inputs

Input

Required

What it is for

Paper PDF

yes

Copied into the run's package/paper/

--citation

no

Authors, year, title, journal: helps the agents identify the paper

--package

no

The authors' replication package (directory or archive) if you already have it; otherwise the package finder searches for it

--data-sources

no, but important

A file describing the data you can access; start from the template (src/ama_mcp/core/data_sources_TEMPLATE.md, also the ama://docs/data_sources_TEMPLATE.md resource)

--data-dir

no

Directories with data the agents may read (passed to the agent CLI)

--supplement

no

The paper's internet appendix or other supplementary files

--headline-anchor

no

JSON listing the outcomes eligible as the headline ({"ia_outcomes": [{"outcome", "location"}]}). The study took this list from an external database; without it the configuration agent picks the main result by its prompt's rule (the paper's first main-results table) and an independent reviewer checks the pick

--forbid

no

Sources no agent may read, e.g. a published comment on or replication of this paper, so the assessment stays independent of it

--data-role-source

no

A third-party source that only the data and reproduction roles may use for data details (e.g. law adoption dates); the other roles are told not to read it

--g0-tol

no

Reproduction-gate tolerance; the study used 0.10

--grid-workers, --grid-mem-gb

no

Parallel worker processes and total memory for the specification grid

--model STEP=MODEL

no

Override the model of one step ('*' for all steps)

--effort LEVEL

no, but important for codex

Reasoning effort of every agent session: low, medium, high, xhigh or max for claude; minimal to ultra for codex (the levels depend on the model). Without it each session uses the CLI's own default; with codex, use high or xhigh, since at low effort the agents give up early on hard steps. ama resume RUN --effort LEVEL changes it for the rest of a run

--stop-after G2

no

Pause once the audited choice space exists (or after any stage), e.g. to inspect it before the grid runs

Data access

The analysis is only as good as the data the agents can reach. If the authors' package ships data and code, the data are used as shipped. If not, the builder rebuilds the data from the original sources you can access: describe them in the data sources file (databases and how to connect, local copies and their paths, what is not available). If the main outcome cannot be built from what is available, the run stops at the package gate as infeasible; that is a recorded result, not a crash.

Respect your data licenses. Agents copy extracts of the data into the run directory (package/, repro/, grid outputs), so a run directory can contain licensed data: do not publish or share it.

Time and cost

In the study, the median paper took about 22 hours from the first agent session to the last and used about 126 million model tokens, most of them cache reads.

Outputs

Everything about a run is in its directory:

File

What

package/

The paper and the replication package (found or rebuilt)

config.yaml, config_review*.md

The main estimate and the independent review

reproduction.md, g0_audit*.md

The reproduction and, after a miss, the reproduction audit

forks.json, audit.json

The analysis choices and their audit; the designated grid and inference

run_grid.py, grid_results.csv, implementation_audit*.md, fix_log.md

The grid and its audit

companion_spec.json, companion_results.csv, companion_audit*.md

The companion grid

paper_measures.json

The headline measures (share supported first), written before the report

grid_survival*.csv, grid_survival*_headline.json

Strict survival and kill power (survival.py)

author_commentary.md, report.md, make_report.py, figures/

The report

gates.log, .frozen.json

Every gate decision; artifact hashes

ama_result.json

The run record, the strict survival headlines and every per-paper measure

orchestrator_state.json, orchestrator.log, agent_logs/, prompts_rendered/

What ran, what each agent was told, and its transcript

A run can end before the report: a gate may decide that the package cannot be built, that the main estimate cannot be estimated, that a reproduction gap stays unexplained, or that the implementation audit does not close. run_status and ama_result.json say where and why.

How the results map to the papers' measures

A run reports two sets of numbers.

  • The headline: the per-paper measures of An AI-Powered Multiverse Analysis of Empirical Finance Research (computed by ama_mcp.metrics; each value comes with its definition). The orchestrator writes them to paper_measures.json after the grid is scored and before the report, so the report leads with them; ama status shows them on one line, and ama results and run_results list them first. They are the share supported (published sign and two-sided p < 0.10 under the designated inference; failed and distinct-estimand specifications left out), the median signed z, the share with the published sign, and whether the main estimate is reproduced (published sign, p ≤ 0.10 under the paper's own inference, and within 25% of the published estimate or, in linear models, less than 0.10 standard deviations of the outcome away). Some of these need facts the pipeline does not write (the claim type, the outcome's standard deviation, the reproduced p-value under the paper's inference); give them in the run's measure_inputs.json (format in src/ama_mcp/metrics.py), otherwise the value is null or rests on a stated assumption. ama_result.json carries every measure of that paper.

  • The strict survival rate of When AI Analysts Disagree (survival.py): over the audited grid, the share of specifications whose estimate has the published sign (sign_consistency_rate) and the share that also has p < 0.05 under the designated inference (survival_rate), plus each choice's kill power (how much survival moves across its options). The companion headline repeats this with the paper's own excluded choices restored. The report gives it after the headline.

Resuming and stopping

The orchestrator records every finished step and every gate decision in orchestrator_state.json. ama resume (or resume_run) continues after a stop, an error, a crash or a pause without repeating finished work; an agent session that was cut off is launched again with a fresh session. A run that a gate stopped stays stopped; --from-stage runs a stage and every later stage again with fresh agents (existing artifacts stay in place). A paused run continues to the end unless ama resume --stop-after STAGE (or resume_run with stop_after) sets a new pause point at a later stage.

Safety

  • Agent sessions run with permission prompts off (Claude Code bypassPermissions; Codex without its sandbox) because they must run long jobs unattended. Use a container or VM.

  • Agents can read and change anything the operating-system user running ama-mcp can. That includes the installed package, so install it where that user cannot write (for example a root-owned environment in the container, with the analysis running as an unprivileged user); otherwise an agent could edit the prompts or gate scripts of its own run.

  • Claude Code sessions start without your MCP servers and auto-memory and, with --safe-mode (used when the installed CLI has it), without your CLAUDE.md files, hooks, skills and plugins. Codex sessions turn off your configured MCP servers one by name and switch off Codex's memories, apps and plugins features; your AGENTS.md files still load. A Codex MCP server whose name has characters other than letters, digits, - and _ cannot be turned off this way, nor can any server when codex mcp list fails; the orchestrator then writes a warning to orchestrator.log. Register this server under a plain name such as ama. Details: docs/CHANGES_FROM_STUDY.md.

  • Keep only the data you are licensed to use inside the environment; run directories may contain licensed data.

  • benchmark/ in a run directory is for held-out material (e.g. a published critique of the paper) that you want to compare against afterwards. Every prompt forbids agents to read it, but nothing enforces this technically: a session could open it. The MCP server's read_run_file refuses to read it. Keep such material out of the environment if independence matters.

Tests

pip install '.[test]'
pytest

The orchestrator tests play every role with a fake agent and run the real gate scripts; no test starts a real agent session unless AMA_LIVE_TESTS is set. tests/test_release.py checks that the repository holds no machine paths, e-mail addresses, tokens or build products. Maintainers can also point AMA_RELEASE_DENYLIST at a private file, kept outside the repository, that lists names which must not appear (one per line).

License

MIT License. See LICENSE.

Citation

If you use this tool, please cite the accompanying papers:

Gao, Ruijiang, and Steven Chong Xiao, 2026, An AI-Powered Multiverse Analysis of Empirical
Finance Research: Evidence from Reused Natural Experiments, Working paper.

Gao, Ruijiang, and Steven Chong Xiao, 2026, When AI Analysts Disagree: Agentic Multiverse
Analysis and the Robustness of Empirical Research, Working paper.
@unpublished{gao2026multiverse,
  author = {Gao, Ruijiang and Xiao, Steven Chong},
  title  = {An {AI}-Powered Multiverse Analysis of Empirical Finance Research: Evidence from Reused Natural Experiments},
  note   = {Working paper},
  year   = {2026}
}

@unpublished{gao2026disagree,
  author = {Gao, Ruijiang and Xiao, Steven Chong},
  title  = {When {AI} Analysts Disagree: Agentic Multiverse Analysis and the Robustness of Empirical Research},
  note   = {Working paper},
  year   = {2026}
}

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      F
      maintenance
      An autonomous software-engineering pipeline for Claude Code that runs on real evidence: codebase intelligence tools, git analytics, deterministic verification rules, and zero LLM-judges-LLM. Every claim traces to a paper and every PR passes its own gates.
      1
      MIT
    • A
      license
      A
      quality
      D
      maintenance
      Enables structured extraction of methods and reproducibility heuristics from academic papers, allowing AI agents to obtain metadata, full text, structured methods, code repository discovery, and a no-clone reproducibility verdict from a paper URL.
      8
      45 PyPI
      MIT