# vibescience-mcp

A **scientific experiment log** (MCP server) for an AI coding/research agent working on ML projects.

It is *not* a lab notebook and *not* Weights & Biases. It stores **reasoning and causal claims**, not training curves, and defines **"confirmed"** as *agreement between a prediction committed **before** the test and the effect observed **after***, on a **fixed diagnostic basis**. That makes your log a queryable **causal map** ("which interventions move which diagnostics, in which direction, how often") plus a **calibration signal** on your own intuition.

Markdown files are the source of truth → open `vault/` in **Obsidian** for graph view, backlinks and tag panels for free. The SQLite index is disposable and rebuilt from markdown at any time.

## Why it's different

| | lab notebook | W&B / MLflow | **vibescience** |
|---|---|---|---|
| stores | free-text notes | loss curves, metrics | **predictions, verdicts, causal claims** |
| "confirmed" means | you say so | — | **predicted direction == observed direction** (computed) |
| negative results | buried | buried | **first-class, ranked to the top of recall** |
| cross-experiment view | none | per-run dashboards | **aggregated causal map + calibration** |

## Install

```bash
cd vibescience-mcp
python -m venv .venv && . .venv/bin/activate
pip install -e .            # add ".[embeddings]" for optional semantic recall, ".[dev]" for tests
```

## Configure in Claude Code / Cursor / Desktop

The server speaks **stdio**. Point your MCP client at it and set the vault path:

```json
{
  "mcpServers": {
    "vibescience": {
      "command": "/root/vibescience-mcp/.venv/bin/python",
      "args": ["-m", "vibescience_mcp.server"],
      "env": {
        "VIBESCIENCE_VAULT": "/root/vibescience-vault",
        "VIBESCIENCE_EMBEDDINGS": "off"
      }
    }
  }
}
```

Or via the Claude Code CLI:

```bash
claude mcp add vibescience -e VIBESCIENCE_VAULT=/root/vibescience-vault \
  -- /root/vibescience-mcp/.venv/bin/python -m vibescience_mcp.server
```

## The scientific loop (enforced by tool descriptions)

1. **`recall`** — *always first* (pre-mortem gate). Refuted/inconclusive matches rank to the **top**, each with a one-line failure reason and the diagnostic delta that killed it. Never re-walk a dead end.
2. **`create_problem`** — frame the open question / failure mode.
3. **`register_diagnostic`** / **`register_intervention`** — diagnostics are a *fixed, comparable basis*, not free text. Adding one is deliberate.
4. **`propose_hypothesis`** — **requires ≥1 `predicted_effect`** on a registered diagnostic *before* testing. The first is the *primary* prediction the verdict keys off. No prediction → rejected.
5. **`start_experiment`** — auto-captures `branch@shortsha` from git HEAD if you omit `git_ref`. References an external W&B/MLflow run; never stores curves.
6. **`record_diagnostics`** — before/after per diagnostic; deltas/directions derived.
7. **`close_experiment`** — **computes** `observed_effects`, `prediction_match` (per-diagnostic + overall) and the `verdict`, propagates status to the hypothesis, and **suggests** (never performs) a commit on a positive match. Record artifacts, headline numbers and dead ends in `artifacts` / `key_numbers` / `do_not_retry` rather than compressing them into `notes`.

**Verdicts.** `supports` · `refutes` · `inconclusive` · `directional_only` (direction matched but a preregistered blocking gate failed) · `null` · `crashed`.

`null` is a **powered null**: the direction nominally matched, a blocking gate carrying a preregistered `mde` failed, and `|delta|` came in *under* that MDE. It is treated as a refutation of the lever at the committed scale — not a near-miss — and ranks with `refutes`. Without it, a sign-only test reports `delta=+0.0003` at permutation `p=0.65` as "the direction matched" and ranks it above `inconclusive`, i.e. the strongest evidence *against* a lever outranks "we don't know". Set `mde` on a gate to get this; omit it and the old two-level behaviour is preserved exactly.

Housekeeping:

- **`doctor`** — reports records whose *status* no longer matches reality: answered problems still `open`, hypotheses rotting in `proposed`/`testing`, experiments left open, near-duplicate experiments started minutes apart (the fingerprint of a double `start_experiment`), runbooks queued on resolved problems, orphan and probable-duplicate tags. Advisory only; every finding names the tool that fixes it.
- **`merge_tags`** — fold a duplicate tag into its canonical form across the whole vault. The source survives as an *alias*, so historical spellings still resolve.
- **`update_problem(add_update=...)`** / **`update_runbook(note=...)`** — dated log entries instead of banners glued onto a description.

Then query your own work:

- **`causal_map(problem_id | tag)`** — aggregated `intervention → Δdiagnostic` subgraph. Optional Obsidian `.canvas` output.
- **`calibration(diagnostic_id | tag | intervention_id)`** — fraction of predictions that matched observation. Where is your intuition wrong?
- **`reindex()`** — rebuild the SQLite index from markdown (idempotent).

A **`vibescience://guide`** resource ships the same workflow to any agent.

## Storage layout

```
vault/
  problems/        hypotheses/     experiments/
  diagnostics/     interventions/  papers/
  _canvas/         # generated Obsidian canvases
  .index.sqlite    # disposable — rebuilt from markdown by reindex()
```

Each entity is one markdown file: YAML frontmatter for structured fields, body for prose + `[[wikilinks]]`. Fully Obsidian-compatible.

## Env vars

| var | default | meaning |
|---|---|---|
| `VIBESCIENCE_VAULT` | `~/vibescience-vault` | vault path (source of truth) |
| `VIBESCIENCE_VAULT_STRICT` | unset | `1` → exit instead of auto-creating a missing vault (catches typo'd paths) |
| `VIBESCIENCE_EMBEDDINGS` | `off` | `on` enables local FastEmbed semantic recall (Phase 2 scaffold) |

`VAULT_DIR` and `VIBESCIENCE_VAULT_DIR` are accepted as aliases (with a warning
on stderr); `--vault PATH` overrides everything. If two of them disagree the
server **exits** rather than guessing. The resolved vault path is printed to
stderr at startup and embedded in the MCP `instructions`, so the agent can
always see which vault it is writing to.

### Running several vaults

One project per vault. Register one server entry per project:

```jsonc
"vibescience":          { "env": { "VIBESCIENCE_VAULT": "/root/vibescience-vault" } },
"vibescience-lightsem": { "env": { "VIBESCIENCE_VAULT": "/root/lightsem-vault"   } }
```

Verify isolation before trusting it — `list_problems` on each server must return
different records. A silently shared vault interleaves records from unrelated
projects, which corrupts the experiment log's role as the system of record.

## Bundled workflow skill

`skills/vibescience-mcp-workflow/SKILL.md` ships the battle-tested procedure for
running the full loop: pair **deep research** (`perplexity_research`) + the
**arxiv** MCP with a real, verdict-computed experiment cycle, then dogfood it
into the vault. It encodes the hard-won pitfalls (research timeouts, making an
experiment able to *refute*, semi-synthetic precision protocols on unlabeled
prod data, the opt-in-vs-blanket deployment guardrail). Load it into any
Hermes/Claude session with `skill_view(name='vibescience-mcp-workflow')`, or read
it directly — it is plain markdown.

## Record IDs

IDs are read by humans — in `ls`, in `[[wikilinks]]`, in error messages. Auto-generated ids are capped at 48 characters **on a word boundary** plus a short hash of the full text, so a shared prefix can never collide; experiments get `exp-NNN-<stem>`, which keeps `ls experiments/` usable as an overview. `start_experiment` is idempotent within a 5-minute window: a retried call returns the existing open experiment instead of minting a second record.

### Migrating an older vault

```bash
python migrate_ids.py <vault_path>            # dry run — shows the plan
python migrate_ids.py <vault_path> --apply    # back the vault up first
```

Renames every over-long id, rewrites all cross-references (`problem_id`, `hypothesis_id`, `parent_experiment_id`, `supersedes`, `paper_refs`) and every wikilink, records the old id in `former_ids` so external references stay traceable, and writes `id_migration.json` as the redirect map. `Store.resolve_id` still finds a record by its old name. Nothing is deleted and no prose is edited — only identifiers change.

## Tests

```bash
./run_tests.sh -q  # 203 tests: verdict math (incl. powered nulls + weak_signal),
                   # prediction gate, full loop, negative-result ranking,
                   # causal_map, calibration, index rebuild, id scheme +
                   # idempotency, doctor, merge_tags, interpretation layer
                   # (lessons, stale negatives, plateau, portfolio,
                   # research_brief), markdown round-trip, MCP stdio smoke
```

## The interpretation layer (2026-08)

Distilled from Prime Intellect's *Measuring Autonomous AI Research* (the gap
between models is experiment hygiene under noise, not idea quality) plus an
autopsy of a real vault: 61 experiments, 39 negative verdicts, **zero recorded
lessons**, 16/17 problems eternally `open`, 30/30 papers never linked to a
hypothesis. A log that records verdicts but never analysis is a junk pile.

- **`interpret_experiment(id, lesson, failure_mode)`** — the act of analysis.
  Negative verdicts require a `failure_mode` (`claim_false / mechanism_wrong /
  test_invalid / noise / execution_bug / recipe_conditional`): a refute from a
  broken capture and a refute of the idea are different objects. `recall`
  surfaces the authored lesson instead of a machine string; `doctor` reports
  closed-but-uninterpreted runs as debt.
- **`reopen_hypothesis(id, reason)`** — a negative result is a fact about
  *(claim × recipe)*, not the claim. `close_experiment` snapshots a
  `recipe_fingerprint` (interventions + provenance hashes); when the recipe
  drifts, a refute flips to `stale_negative` — flagged re-testable, history
  untouched, the re-test still goes through `supersedes` + a fresh prediction.
- **`progress(problem_id)` + `synthesize_problem`** — the compiled trajectory
  (primary-diagnostic movement, verdict counts, plateau, interpretation debt)
  and the authored current-state-of-knowledge that turns a chronology into a
  knowledge base.
- **Noise floors** — `register_diagnostic(..., noise_sigma, noise_source=
  'measured')` makes the verdict engine demote a directionally-correct delta
  inside `k*sigma` to **`weak_signal`**: a candidate to re-draw, neither a win
  to deploy nor a loss to discard. The top autonomous-research runs measured
  their own noise; the weak ones shipped draw luck.
- **Ambition economics** — hypotheses declare `ambition_class`
  (`tuning / mechanism / reframe`) and `gap_closure_estimate` against the
  problem's declared `target`; experiments carry `gpu_hours`. `portfolio()`
  shows where the compute went per class, and a **plateaued** problem (last 5
  closed runs, no `supports`) rejects further `tuning` without an
  `override_reason` — the crumb trap made expensive.
- **`research_brief(problem_id)`** — compiles state + gap + dead ideas (with
  lessons) + rule-outs + the already-read paper **exclusion list** into a ready
  deep-research query, so every literature sweep is incremental by
  construction. The literature is the edge a sandboxed harness doesn't have.

## Phasing

- **Phase 1 (this):** schema + lifecycle + `recall` + `causal_map` + `calibration` over markdown, SQLite index, Obsidian compatibility.
- **Phase 2 (scaffolded, not built):** hold papers in context — fetch arXiv full text, chunk, embed, and let `recall` pull relevant passages in. The `Paper` schema is ready.

**Non-goals:** custom graph visualization (Obsidian does it), web UI, autonomous experiment execution, raw metric-curve storage (reference W&B/MLflow), auto-committing to git.

## License

MIT