vibescience-mcp

A scientific experiment log (MCP server) for an AI coding/research agent working on ML projects.

It is not a lab notebook and not Weights & Biases. It stores reasoning and causal claims, not training curves, and defines "confirmed" as agreement between a prediction committed before the test and the effect observed after, on a fixed diagnostic basis. That makes your log a queryable causal map ("which interventions move which diagnostics, in which direction, how often") plus a calibration signal on your own intuition.

Markdown files are the source of truth → open vault/ in Obsidian for graph view, backlinks and tag panels for free. The SQLite index is disposable and rebuilt from markdown at any time.

Why it's different

Related MCP server: repro-mcp

Install

cd vibescience-mcp
python -m venv .venv && . .venv/bin/activate
pip install -e .            # add ".[embeddings]" for optional semantic recall, ".[dev]" for tests

Configure in Claude Code / Cursor / Desktop

The server speaks stdio. Point your MCP client at it and set the vault path:

{
  "mcpServers": {
    "vibescience": {
      "command": "/root/vibescience-mcp/.venv/bin/python",
      "args": ["-m", "vibescience_mcp.server"],
      "env": {
        "VIBESCIENCE_VAULT": "/root/vibescience-vault",
        "VIBESCIENCE_EMBEDDINGS": "off"
      }
    }
  }
}

Or via the Claude Code CLI:

claude mcp add vibescience -e VIBESCIENCE_VAULT=/root/vibescience-vault \
  -- /root/vibescience-mcp/.venv/bin/python -m vibescience_mcp.server

The scientific loop (enforced by tool descriptions)

1. recall — always first (pre-mortem gate). Refuted/inconclusive matches rank to the top, each with a one-line failure reason and the diagnostic delta that killed it. Never re-walk a dead end.

2. create_problem — frame the open question / failure mode.

3. register_diagnostic / register_intervention — diagnostics are a fixed, comparable basis, not free text. Adding one is deliberate.

4. propose_hypothesis — requires ≥1 predicted_effect on a registered diagnostic before testing. The first is the primary prediction the verdict keys off. No prediction → rejected.

5. start_experiment — auto-captures branch@shortsha from git HEAD if you omit git_ref. References an external W&B/MLflow run; never stores curves.

6. record_diagnostics — before/after per diagnostic; deltas/directions derived.

7. close_experiment — computes observed_effects, prediction_match (per-diagnostic + overall) and the verdict, propagates status to the hypothesis, and suggests (never performs) a commit on a positive match. Record artifacts, headline numbers and dead ends in artifacts / key_numbers / do_not_retry rather than compressing them into notes.

Verdicts. supports · refutes · inconclusive · directional_only (direction matched but a preregistered blocking gate failed) · null · crashed.

null is a powered null: the direction nominally matched, a blocking gate carrying a preregistered mde failed, and |delta| came in under that MDE. It is treated as a refutation of the lever at the committed scale — not a near-miss — and ranks with refutes. Without it, a sign-only test reports delta=+0.0003 at permutation p=0.65 as "the direction matched" and ranks it above inconclusive, i.e. the strongest evidence against a lever outranks "we don't know". Set mde on a gate to get this; omit it and the old two-level behaviour is preserved exactly.

Housekeeping:

• doctor — reports records whose status no longer matches reality: answered problems still open, hypotheses rotting in proposed/testing, experiments left open, near-duplicate experiments started minutes apart (the fingerprint of a double start_experiment), runbooks queued on resolved problems, orphan and probable-duplicate tags. Advisory only; every finding names the tool that fixes it.

• merge_tags — fold a duplicate tag into its canonical form across the whole vault. The source survives as an alias, so historical spellings still resolve.

• update_problem(add_update=...) / update_runbook(note=...) — dated log entries instead of banners glued onto a description.

Then query your own work:

• causal_map(problem_id | tag) — aggregated intervention → Δdiagnostic subgraph. Optional Obsidian .canvas output.

• calibration(diagnostic_id | tag | intervention_id) — fraction of predictions that matched observation. Where is your intuition wrong?

• reindex() — rebuild the SQLite index from markdown (idempotent).

A vibescience://guide resource ships the same workflow to any agent.

Storage layout

vault/
  problems/        hypotheses/     experiments/
  diagnostics/     interventions/  papers/
  _canvas/         # generated Obsidian canvases
  .index.sqlite    # disposable — rebuilt from markdown by reindex()

Each entity is one markdown file: YAML frontmatter for structured fields, body for prose + [[wikilinks]]. Fully Obsidian-compatible.

Env vars

VAULT_DIR and VIBESCIENCE_VAULT_DIR are accepted as aliases (with a warning on stderr); --vault PATH overrides everything. If two of them disagree the server exits rather than guessing. The resolved vault path is printed to stderr at startup and embedded in the MCP instructions, so the agent can always see which vault it is writing to.

Running several vaults

One project per vault. Register one server entry per project:

"vibescience":          { "env": { "VIBESCIENCE_VAULT": "/root/vibescience-vault" } },
"vibescience-lightsem": { "env": { "VIBESCIENCE_VAULT": "/root/lightsem-vault"   } }

Verify isolation before trusting it — list_problems on each server must return different records. A silently shared vault interleaves records from unrelated projects, which corrupts the experiment log's role as the system of record.

Bundled workflow skill

skills/vibescience-mcp-workflow/SKILL.md ships the battle-tested procedure for running the full loop: pair deep research (perplexity_research) + the arxiv MCP with a real, verdict-computed experiment cycle, then dogfood it into the vault. It encodes the hard-won pitfalls (research timeouts, making an experiment able to refute, semi-synthetic precision protocols on unlabeled prod data, the opt-in-vs-blanket deployment guardrail). Load it into any Hermes/Claude session with skill_view(name='vibescience-mcp-workflow'), or read it directly — it is plain markdown.

Record IDs

IDs are read by humans — in ls, in [[wikilinks]], in error messages. Auto-generated ids are capped at 48 characters on a word boundary plus a short hash of the full text, so a shared prefix can never collide; experiments get exp-NNN-<stem>, which keeps ls experiments/ usable as an overview. start_experiment is idempotent within a 5-minute window: a retried call returns the existing open experiment instead of minting a second record.

Migrating an older vault

python migrate_ids.py <vault_path>            # dry run — shows the plan
python migrate_ids.py <vault_path> --apply    # back the vault up first

Renames every over-long id, rewrites all cross-references (problem_id, hypothesis_id, parent_experiment_id, supersedes, paper_refs) and every wikilink, records the old id in former_ids so external references stay traceable, and writes id_migration.json as the redirect map. Store.resolve_id still finds a record by its old name. Nothing is deleted and no prose is edited — only identifiers change.

Tests

./run_tests.sh -q  # 203 tests: verdict math (incl. powered nulls + weak_signal),
                   # prediction gate, full loop, negative-result ranking,
                   # causal_map, calibration, index rebuild, id scheme +
                   # idempotency, doctor, merge_tags, interpretation layer
                   # (lessons, stale negatives, plateau, portfolio,
                   # research_brief), markdown round-trip, MCP stdio smoke

The interpretation layer (2026-08)

Distilled from Prime Intellect's Measuring Autonomous AI Research (the gap between models is experiment hygiene under noise, not idea quality) plus an autopsy of a real vault: 61 experiments, 39 negative verdicts, zero recorded lessons, 16/17 problems eternally open, 30/30 papers never linked to a hypothesis. A log that records verdicts but never analysis is a junk pile.

• interpret_experiment(id, lesson, failure_mode) — the act of analysis. Negative verdicts require a failure_mode (claim_false / mechanism_wrong / test_invalid / noise / execution_bug / recipe_conditional): a refute from a broken capture and a refute of the idea are different objects. recall surfaces the authored lesson instead of a machine string; doctor reports closed-but-uninterpreted runs as debt.

• reopen_hypothesis(id, reason) — a negative result is a fact about (claim × recipe), not the claim. close_experiment snapshots a recipe_fingerprint (interventions + provenance hashes); when the recipe drifts, a refute flips to stale_negative — flagged re-testable, history untouched, the re-test still goes through supersedes + a fresh prediction.

• progress(problem_id) + synthesize_problem — the compiled trajectory (primary-diagnostic movement, verdict counts, plateau, interpretation debt) and the authored current-state-of-knowledge that turns a chronology into a knowledge base.

• Noise floors — register_diagnostic(..., noise_sigma, noise_source= 'measured') makes the verdict engine demote a directionally-correct delta inside k*sigma to weak_signal: a candidate to re-draw, neither a win to deploy nor a loss to discard. The top autonomous-research runs measured their own noise; the weak ones shipped draw luck.

• Ambition economics — hypotheses declare ambition_class (tuning / mechanism / reframe) and gap_closure_estimate against the problem's declared target; experiments carry gpu_hours. portfolio() shows where the compute went per class, and a plateaued problem (last 5 closed runs, no supports) rejects further tuning without an override_reason — the crumb trap made expensive.

• research_brief(problem_id) — compiles state + gap + dead ideas (with lessons) + rule-outs + the already-read paper exclusion list into a ready deep-research query, so every literature sweep is incremental by construction. The literature is the edge a sandboxed harness doesn't have.

Phasing

• Phase 1 (this): schema + lifecycle + recall + causal_map + calibration over markdown, SQLite index, Obsidian compatibility.

• Phase 2 (scaffolded, not built): hold papers in context — fetch arXiv full text, chunk, embed, and let recall pull relevant passages in. The Paper schema is ready.

Non-goals: custom graph visualization (Obsidian does it), web UI, autonomous experiment execution, raw metric-curve storage (reference W&B/MLflow), auto-committing to git.

License

MIT