Skip to main content
Glama
README.md
# agentic_multiverse_analysis

Agentic multiverse analysis of an empirical paper, as an MCP server (`ama-mcp`) and a command line (`ama`).

`ama-mcp` runs an **agentic multiverse analysis (AMA)** of one published empirical paper and
reports how much of the paper's defensible specification space keeps its main conclusion. It
is the pipeline used for the two accompanying papers (see [Citation](#citation)), packaged as an
MCP server (`ama-mcp`) and a command line (`ama`). Every role in the pipeline runs as a fresh
agent session through **your own Claude Code (`claude`) or OpenAI Codex (`codex`) CLI login**.

> **Safety first.** The agent sessions run code from replication packages, install packages,
> download data and browse the web, with permission prompts turned off. **Run `ama-mcp` inside
> a container or a virtual machine** that holds only what the analysis needs (the paper, the
> data you are licensed to use, the agent CLI login). Do not run it on a machine whose files or
> credentials you would not hand to an unattended agent. See [Safety](#safety).

## What AMA does

You give it a paper. It then:

1. **Finds the authors' replication package** (journal archives, data repositories, author
   pages). If there is no complete package, an agent **rebuilds the data** from the original
   sources you can access, following the paper's own description, and an independent
   reviewer checks every sample rule and reported sample size against the paper.
2. **Pins down the main estimate.** One agent writes a configuration naming the paper's
   headline coefficient, its table location and the claimed sign; another agent checks it
   against the paper.
3. **Reproduces the main estimate** with its own code. A gate script compares the result with
   the published number (within 10% by default). On a miss, an independent auditor looks for
   implementation errors before the run may continue.
4. **Lists the analysis choices.** One agent lists every choice the analysis makes (sample,
   variable definitions, controls, fixed effects, estimator, inference, ...) and the
   defensible alternatives. A *different* agent judges each option against a fixed rubric and
   designs a grid of 100 to 250 specifications, and fixes the inference method, **before any
   alternative estimate exists**.
5. **Runs the grid.** An implementer writes and runs the grid; an independent auditor reads
   the code line by line and recomputes cells; fix rounds continue until the auditor finds no
   blocking problem.
6. **Measures what the auditor's exclusions cost the paper** (a "companion" grid with the
   paper's own choices restored).
7. **Reports.** The papers' measures are computed by scripts; an agent collects what the authors
   say about the decisive choices, and a reporter writes a report whose every number is
   regenerated by a script and checked byte for byte.

Three rules make the output worth reading: no agent checks its own work; the space of choices
is judged before any estimate exists; every step ends at a gate decided by a program or by an
independent agent, never by the agent whose work is gated. Gate decisions are appended to
`gates.log`, and artifacts are hashed when a gate passes so a later edit is detected.

The prompts, gate scripts and orchestration contract in `src/ama_mcp/core/` are the ones the
study ran, changed only where a public user's machine needs it. Every change is listed, with
its reason, in [`docs/CHANGES_FROM_STUDY.md`](docs/CHANGES_FROM_STUDY.md).

## Requirements

- Linux or macOS, Python 3.10 or newer.
- At least one agent CLI, installed and logged in:
  - **Claude Code**: `claude` on your `PATH`; log in once interactively (`claude`, then
    `/login`) or set an API key as Claude Code documents.
  - **OpenAI Codex CLI**: `codex` on your `PATH`; `codex login`.
- A Python environment for the agents' analysis code. The agents write Python by default; a
  typical environment has `pandas`, `numpy`, `statsmodels`, `linearmodels`, `pyfixest`,
  `wildboottest`, `pyreadstat`, `pyarrow` and `matplotlib`. Pass it with `--analysis-python`.
  Stata or R are used only if a replication package needs them and they are installed.
- Access to the data the paper uses, unless the authors' package ships it (see
  [Data access](#data-access)).

## Install

```bash
python -m venv .venv
.venv/bin/pip install .            # from a clone of this repository
.venv/bin/ama doctor               # checks the core files, Python packages and agent CLIs
```

`ama doctor` reports whether each CLI is installed and logged in, whether the runs directory
is writable, and whether it is running inside a container.

## Use it from Claude Code or Codex (MCP)

Register the server once. Use the absolute path of the `ama-mcp` executable in your
environment if it is not on your `PATH` (`ama-mcp --help` prints both registrations with that
path filled in).

**Claude Code**

```bash
claude mcp add --scope user ama -e AMA_RUNS_ROOT="$HOME/ama-runs" -- /path/to/.venv/bin/ama-mcp
```

**Codex** (`~/.codex/config.toml`)

```toml
[mcp_servers.ama]
command = "/path/to/.venv/bin/ama-mcp"
env = { AMA_RUNS_ROOT = "/path/to/ama-runs" }
```

Then ask your agent, for example: *"Use the ama tools to run an agentic multiverse analysis of
~/papers/paper.pdf with the claude backend."* The server also ships an MCP prompt,
`run_analysis`, that walks an agent through the steps.

The backend that runs the analysis is independent of the client you talk to: a Codex session
can start a run on the `claude` backend and the other way round.

| Tool | What it does |
|---|---|
| `check_backends` | Is `claude` / `codex` installed and logged in? (never returns credentials) |
| `analyze_paper` | Creates a run directory and starts the analysis in the background; returns a `run_id`. Takes the options of `ama run`, including `effort` |
| `run_status` | Status, current stage, gate outcomes so far, the headline once it exists, the last log lines |
| `run_results` | `ama_result.json`, the headline (share supported first), the strict survival rate, all measures, a report excerpt |
| `list_runs` | All runs, newest first |
| `stop_run` / `resume_run` | Stop a run; continue it from where it stopped |
| `read_run_file` | Read a text file of a run (the held-out `benchmark/` folder is never readable) |

Resources: the pipeline documents (`ama://docs/...`) and the 18 role prompt templates
(`ama://prompts/...`).

## Use it from the command line

```bash
ama run paper.pdf --backend claude \
    --citation "Author and Author (2015), Title, Journal" \
    --data-sources my_data_sources.md --analysis-python /path/to/analysis-env/bin/python
ama status                 # all runs
ama status <run_id>        # one run: stage, gates, headline, last log lines
ama results <run_id>       # headline, strict survival, measures (--all, --definitions), report excerpt
ama stop <run_id>
ama resume <run_id>        # continue after a stop, an error or a pause
ama doctor
```

`ama run` starts the analysis in the background and returns; add `--foreground` to keep it
attached to the terminal. Runs live in `$AMA_RUNS_ROOT` (default `~/ama-runs`). The
orchestrator can also be driven directly: `python -m ama_mcp.orchestrator run --run-dir RUN`.

## Inputs

| Input | Required | What it is for |
|---|---|---|
| Paper PDF | yes | Copied into the run's `package/paper/` |
| `--citation` | no | Authors, year, title, journal: helps the agents identify the paper |
| `--package` | no | The authors' replication package (directory or archive) if you already have it; otherwise the package finder searches for it |
| `--data-sources` | no, but important | A file describing the data you can access; start from the template (`src/ama_mcp/core/data_sources_TEMPLATE.md`, also the `ama://docs/data_sources_TEMPLATE.md` resource) |
| `--data-dir` | no | Directories with data the agents may read (passed to the agent CLI) |
| `--supplement` | no | The paper's internet appendix or other supplementary files |
| `--headline-anchor` | no | JSON listing the outcomes eligible as the headline (`{"ia_outcomes": [{"outcome", "location"}]}`). The study took this list from an external database; without it the configuration agent picks the main result by its prompt's rule (the paper's first main-results table) and an independent reviewer checks the pick |
| `--forbid` | no | Sources no agent may read, e.g. a published comment on or replication of this paper, so the assessment stays independent of it |
| `--data-role-source` | no | A third-party source that only the data and reproduction roles may use for data details (e.g. law adoption dates); the other roles are told not to read it |
| `--g0-tol` | no | Reproduction-gate tolerance; the study used 0.10 |
| `--grid-workers`, `--grid-mem-gb` | no | Parallel worker processes and total memory for the specification grid |
| `--model STEP=MODEL` | no | Override the model of one step (`'*'` for all steps) |
| `--effort LEVEL` | no, but important for `codex` | Reasoning effort of every agent session: `low`, `medium`, `high`, `xhigh` or `max` for `claude`; `minimal` to `ultra` for `codex` (the levels depend on the model). Without it each session uses the CLI's own default; with `codex`, use `high` or `xhigh`, since at low effort the agents give up early on hard steps. `ama resume RUN --effort LEVEL` changes it for the rest of a run |
| `--stop-after G2` | no | Pause once the audited choice space exists (or after any stage), e.g. to inspect it before the grid runs |

## Data access

The analysis is only as good as the data the agents can reach. If the authors' package ships
data and code, the data are used as shipped. If not, the builder rebuilds the data from the
original sources **you** can access: describe them in the data sources file (databases and how
to connect, local copies and their paths, what is *not* available). If the main outcome cannot
be built from what is available, the run stops at the package gate as *infeasible*; that is a
recorded result, not a crash.

Respect your data licenses. Agents copy extracts of the data into the run directory
(`package/`, `repro/`, grid outputs), so a run directory can contain licensed data: do not
publish or share it.

## Time and cost

In the study, the median paper took about 22 hours from the first agent session to the last and used
about 126 million model tokens, most of them cache reads.

## Outputs

Everything about a run is in its directory:

| File | What |
|---|---|
| `package/` | The paper and the replication package (found or rebuilt) |
| `config.yaml`, `config_review*.md` | The main estimate and the independent review |
| `reproduction.md`, `g0_audit*.md` | The reproduction and, after a miss, the reproduction audit |
| `forks.json`, `audit.json` | The analysis choices and their audit; the designated grid and inference |
| `run_grid.py`, `grid_results.csv`, `implementation_audit*.md`, `fix_log.md` | The grid and its audit |
| `companion_spec.json`, `companion_results.csv`, `companion_audit*.md` | The companion grid |
| `paper_measures.json` | The headline measures (share supported first), written before the report |
| `grid_survival*.csv`, `grid_survival*_headline.json` | Strict survival and kill power (`survival.py`) |
| `author_commentary.md`, `report.md`, `make_report.py`, `figures/` | The report |
| `gates.log`, `.frozen.json` | Every gate decision; artifact hashes |
| `ama_result.json` | The run record, the strict survival headlines and every per-paper measure |
| `orchestrator_state.json`, `orchestrator.log`, `agent_logs/`, `prompts_rendered/` | What ran, what each agent was told, and its transcript |

A run can end before the report: a gate may decide that the package cannot be built, that the
main estimate cannot be estimated, that a reproduction gap stays unexplained, or that the
implementation audit does not close. `run_status` and `ama_result.json` say where and why.

## How the results map to the papers' measures

A run reports two sets of numbers.

- **The headline: the per-paper measures of *An AI-Powered Multiverse Analysis of Empirical Finance Research*** (computed by `ama_mcp.metrics`; each value
  comes with its definition). The orchestrator writes them to `paper_measures.json` after the
  grid is scored and before the report, so the report leads with them; `ama status` shows them
  on one line, and `ama results` and `run_results` list them first. They are the **share
  supported** (published sign and two-sided p < 0.10 under the designated inference; failed and
  distinct-estimand specifications left out), the **median signed z**, the **share with the
  published sign**, and whether the main estimate is **reproduced** (published sign, p ≤ 0.10
  under the paper's own inference, and within 25% of the published estimate or, in linear
  models, less than 0.10 standard deviations of the outcome away). Some of these need facts the
  pipeline does not write (the claim type, the outcome's standard deviation, the reproduced
  p-value under the paper's inference); give them in the run's `measure_inputs.json` (format in
  `src/ama_mcp/metrics.py`), otherwise the value is null or rests on a stated assumption.
  `ama_result.json` carries every measure of that paper.
- **The strict survival rate of *When AI Analysts Disagree*** (`survival.py`): over the audited grid, the share of
  specifications whose estimate has the published sign (`sign_consistency_rate`) and the share
  that also has p < 0.05 under the designated inference (`survival_rate`), plus each choice's
  *kill power* (how much survival moves across its options). The companion headline repeats
  this with the paper's own excluded choices restored. The report gives it after the headline.

## Resuming and stopping

The orchestrator records every finished step and every gate decision in
`orchestrator_state.json`. `ama resume` (or `resume_run`) continues after a stop, an error, a
crash or a pause without repeating finished work; an agent session that was cut off is
launched again with a fresh session. A run that a gate stopped stays stopped; `--from-stage`
runs a stage and every later stage again with fresh agents (existing artifacts stay in place).
A paused run continues to the end unless `ama resume --stop-after STAGE` (or `resume_run` with
`stop_after`) sets a new pause point at a later stage.

## Safety

- Agent sessions run with permission prompts off (Claude Code `bypassPermissions`; Codex
  without its sandbox) because they must run long jobs unattended. Use a container or VM.
- Agents can read and change anything the operating-system user running `ama-mcp` can. That
  includes the installed package, so install it where that user cannot write (for example a
  root-owned environment in the container, with the analysis running as an unprivileged
  user); otherwise an agent could edit the prompts or gate scripts of its own run.
- Claude Code sessions start without your MCP servers and auto-memory and, with `--safe-mode`
  (used when the installed CLI has it), without your CLAUDE.md files, hooks, skills and
  plugins. Codex sessions turn off your configured MCP servers one by name and switch off
  Codex's memories, apps and plugins features; your `AGENTS.md` files still load. A Codex MCP
  server whose name has characters other than letters, digits, `-` and `_` cannot be turned
  off this way, nor can any server when `codex mcp list` fails; the orchestrator then writes a
  warning to `orchestrator.log`. Register this server under a plain name such as `ama`.
  Details: [`docs/CHANGES_FROM_STUDY.md`](docs/CHANGES_FROM_STUDY.md).
- Keep only the data you are licensed to use inside the environment; run directories may
  contain licensed data.
- `benchmark/` in a run directory is for held-out material (e.g. a published critique of the
  paper) that you want to compare against afterwards. Every prompt forbids agents to read it,
  but nothing enforces this technically: a session could open it. The MCP server's
  `read_run_file` refuses to read it. Keep such material out of the environment if
  independence matters.

## Tests

```bash
pip install '.[test]'
pytest
```

The orchestrator tests play every role with a fake agent and run the real gate scripts; no
test starts a real agent session unless `AMA_LIVE_TESTS` is set. `tests/test_release.py`
checks that the repository holds no machine paths, e-mail addresses, tokens or build products.
Maintainers can also point `AMA_RELEASE_DENYLIST` at a private file, kept outside the
repository, that lists names which must not appear (one per line).

## License

MIT License. See [LICENSE](LICENSE).

## Citation

If you use this tool, please cite the accompanying papers:

```
Gao, Ruijiang, and Steven Chong Xiao, 2026, An AI-Powered Multiverse Analysis of Empirical
Finance Research: Evidence from Reused Natural Experiments, Working paper.

Gao, Ruijiang, and Steven Chong Xiao, 2026, When AI Analysts Disagree: Agentic Multiverse
Analysis and the Robustness of Empirical Research, Working paper.
```

```bibtex
@unpublished{gao2026multiverse,
  author = {Gao, Ruijiang and Xiao, Steven Chong},
  title  = {An {AI}-Powered Multiverse Analysis of Empirical Finance Research: Evidence from Reused Natural Experiments},
  note   = {Working paper},
  year   = {2026}
}

@unpublished{gao2026disagree,
  author = {Gao, Ruijiang and Xiao, Steven Chong},
  title  = {When {AI} Analysts Disagree: Agentic Multiverse Analysis and the Robustness of Empirical Research},
  note   = {Working paper},
  year   = {2026}
}
```