Skip to main content
Glama
vibescience-mcp

vibescience-mcp

by AntonIXO

vibescience-mcp

A scientific experiment log (MCP server) for an AI coding/research agent working on ML projects.

It is not a lab notebook and not Weights & Biases. It stores reasoning and causal claims, not training curves, and defines "confirmed" as agreement between a prediction committed before the test and the effect observed after, on a fixed diagnostic basis. That makes your log a queryable causal map ("which interventions move which diagnostics, in which direction, how often") plus a calibration signal on your own intuition.

Markdown files are the source of truth → open vault/ in Obsidian for graph view, backlinks and tag panels for free. The SQLite index is disposable and rebuilt from markdown at any time.

Why it's different

lab notebook

W&B / MLflow

vibescience

stores

free-text notes

loss curves, metrics

predictions, verdicts, causal claims

"confirmed" means

you say so

—

predicted direction == observed direction (computed)

negative results

buried

buried

first-class, ranked to the top of recall

cross-experiment view

none

per-run dashboards

aggregated causal map + calibration

Related MCP server: hive-exp

Install

cd vibescience-mcp
python -m venv .venv && . .venv/bin/activate
pip install -e .            # add ".[embeddings]" for optional semantic recall, ".[dev]" for tests

Configure in Claude Code / Cursor / Desktop

The server speaks stdio. Point your MCP client at it and set the vault path:

{
  "mcpServers": {
    "vibescience": {
      "command": "/root/vibescience-mcp/.venv/bin/python",
      "args": ["-m", "vibescience_mcp.server"],
      "env": {
        "VIBESCIENCE_VAULT": "/root/vibescience-vault",
        "VIBESCIENCE_EMBEDDINGS": "off"
      }
    }
  }
}

Or via the Claude Code CLI:

claude mcp add vibescience -e VIBESCIENCE_VAULT=/root/vibescience-vault \
  -- /root/vibescience-mcp/.venv/bin/python -m vibescience_mcp.server

The scientific loop (enforced by tool descriptions)

  1. recall — always first (pre-mortem gate). Refuted/inconclusive matches rank to the top, each with a one-line failure reason and the diagnostic delta that killed it. Never re-walk a dead end.

  2. create_problem — frame the open question / failure mode.

  3. register_diagnostic / register_intervention — diagnostics are a fixed, comparable basis, not free text. Adding one is deliberate.

  4. propose_hypothesis — requires ≥1 predicted_effect on a registered diagnostic before testing. The first is the primary prediction the verdict keys off. No prediction → rejected.

  5. start_experiment — auto-captures branch@shortsha from git HEAD if you omit git_ref. References an external W&B/MLflow run; never stores curves.

  6. record_diagnostics — before/after per diagnostic; deltas/directions derived.

  7. close_experiment — computes observed_effects, prediction_match (per-diagnostic + overall) and the verdict, propagates status to the hypothesis, and suggests (never performs) a commit on a positive match. Record artifacts, headline numbers and dead ends in artifacts / key_numbers / do_not_retry rather than compressing them into notes.

Verdicts. supports · refutes · inconclusive · directional_only (direction matched but a preregistered blocking gate failed) · null · crashed.

null is a powered null: the direction nominally matched, a blocking gate carrying a preregistered mde failed, and |delta| came in under that MDE. It is treated as a refutation of the lever at the committed scale — not a near-miss — and ranks with refutes. Without it, a sign-only test reports delta=+0.0003 at permutation p=0.65 as "the direction matched" and ranks it above inconclusive, i.e. the strongest evidence against a lever outranks "we don't know". Set mde on a gate to get this; omit it and the old two-level behaviour is preserved exactly.

Housekeeping:

  • doctor — reports records whose status no longer matches reality: answered problems still open, hypotheses rotting in proposed/testing, experiments left open, near-duplicate experiments started minutes apart (the fingerprint of a double start_experiment), runbooks queued on resolved problems, orphan and probable-duplicate tags. Advisory only; every finding names the tool that fixes it.

  • merge_tags — fold a duplicate tag into its canonical form across the whole vault. The source survives as an alias, so historical spellings still resolve.

  • update_problem(add_update=...) / update_runbook(note=...) — dated log entries instead of banners glued onto a description.

Then query your own work:

  • causal_map(problem_id | tag) — aggregated intervention → Δdiagnostic subgraph. Optional Obsidian .canvas output.

  • calibration(diagnostic_id | tag | intervention_id) — fraction of predictions that matched observation. Where is your intuition wrong?

  • reindex() — rebuild the SQLite index from markdown (idempotent).

A vibescience://guide resource ships the same workflow to any agent.

Storage layout

vault/
  problems/        hypotheses/     experiments/
  diagnostics/     interventions/  papers/
  _canvas/         # generated Obsidian canvases
  .index.sqlite    # disposable — rebuilt from markdown by reindex()

Each entity is one markdown file: YAML frontmatter for structured fields, body for prose + [[wikilinks]]. Fully Obsidian-compatible.

Env vars

var

default

meaning

VIBESCIENCE_VAULT

~/vibescience-vault

vault path (source of truth)

VIBESCIENCE_VAULT_STRICT

unset

1 → exit instead of auto-creating a missing vault (catches typo'd paths)

VIBESCIENCE_EMBEDDINGS

off

on enables local FastEmbed semantic recall (Phase 2 scaffold)

VAULT_DIR and VIBESCIENCE_VAULT_DIR are accepted as aliases (with a warning on stderr); --vault PATH overrides everything. If two of them disagree the server exits rather than guessing. The resolved vault path is printed to stderr at startup and embedded in the MCP instructions, so the agent can always see which vault it is writing to.

Running several vaults

One project per vault. Register one server entry per project:

"vibescience":          { "env": { "VIBESCIENCE_VAULT": "/root/vibescience-vault" } },
"vibescience-lightsem": { "env": { "VIBESCIENCE_VAULT": "/root/lightsem-vault"   } }

Verify isolation before trusting it — list_problems on each server must return different records. A silently shared vault interleaves records from unrelated projects, which corrupts the experiment log's role as the system of record.

Bundled workflow skill

skills/vibescience-mcp-workflow/SKILL.md ships the battle-tested procedure for running the full loop: pair deep research (perplexity_research) + the arxiv MCP with a real, verdict-computed experiment cycle, then dogfood it into the vault. It encodes the hard-won pitfalls (research timeouts, making an experiment able to refute, semi-synthetic precision protocols on unlabeled prod data, the opt-in-vs-blanket deployment guardrail). Load it into any Hermes/Claude session with skill_view(name='vibescience-mcp-workflow'), or read it directly — it is plain markdown.

Record IDs

IDs are read by humans — in ls, in [[wikilinks]], in error messages. Auto-generated ids are capped at 48 characters on a word boundary plus a short hash of the full text, so a shared prefix can never collide; experiments get exp-NNN-<stem>, which keeps ls experiments/ usable as an overview. start_experiment is idempotent within a 5-minute window: a retried call returns the existing open experiment instead of minting a second record.

Migrating an older vault

python migrate_ids.py <vault_path>            # dry run — shows the plan
python migrate_ids.py <vault_path> --apply    # back the vault up first

Renames every over-long id, rewrites all cross-references (problem_id, hypothesis_id, parent_experiment_id, supersedes, paper_refs) and every wikilink, records the old id in former_ids so external references stay traceable, and writes id_migration.json as the redirect map. Store.resolve_id still finds a record by its old name. Nothing is deleted and no prose is edited — only identifiers change.

Tests

./run_tests.sh -q  # 203 tests: verdict math (incl. powered nulls + weak_signal),
                   # prediction gate, full loop, negative-result ranking,
                   # causal_map, calibration, index rebuild, id scheme +
                   # idempotency, doctor, merge_tags, interpretation layer
                   # (lessons, stale negatives, plateau, portfolio,
                   # research_brief), markdown round-trip, MCP stdio smoke

The interpretation layer (2026-08)

Distilled from Prime Intellect's Measuring Autonomous AI Research (the gap between models is experiment hygiene under noise, not idea quality) plus an autopsy of a real vault: 61 experiments, 39 negative verdicts, zero recorded lessons, 16/17 problems eternally open, 30/30 papers never linked to a hypothesis. A log that records verdicts but never analysis is a junk pile.

  • interpret_experiment(id, lesson, failure_mode) — the act of analysis. Negative verdicts require a failure_mode (claim_false / mechanism_wrong / test_invalid / noise / execution_bug / recipe_conditional): a refute from a broken capture and a refute of the idea are different objects. recall surfaces the authored lesson instead of a machine string; doctor reports closed-but-uninterpreted runs as debt.

  • reopen_hypothesis(id, reason) — a negative result is a fact about (claim × recipe), not the claim. close_experiment snapshots a recipe_fingerprint (interventions + provenance hashes); when the recipe drifts, a refute flips to stale_negative — flagged re-testable, history untouched, the re-test still goes through supersedes + a fresh prediction.

  • progress(problem_id) + synthesize_problem — the compiled trajectory (primary-diagnostic movement, verdict counts, plateau, interpretation debt) and the authored current-state-of-knowledge that turns a chronology into a knowledge base.

  • Noise floors — register_diagnostic(..., noise_sigma, noise_source= 'measured') makes the verdict engine demote a directionally-correct delta inside k*sigma to weak_signal: a candidate to re-draw, neither a win to deploy nor a loss to discard. The top autonomous-research runs measured their own noise; the weak ones shipped draw luck.

  • Ambition economics — hypotheses declare ambition_class (tuning / mechanism / reframe) and gap_closure_estimate against the problem's declared target; experiments carry gpu_hours. portfolio() shows where the compute went per class, and a plateaued problem (last 5 closed runs, no supports) rejects further tuning without an override_reason — the crumb trap made expensive.

  • research_brief(problem_id) — compiles state + gap + dead ideas (with lessons) + rule-outs + the already-read paper exclusion list into a ready deep-research query, so every literature sweep is incremental by construction. The literature is the edge a sandboxed harness doesn't have.

Phasing

  • Phase 1 (this): schema + lifecycle + recall + causal_map + calibration over markdown, SQLite index, Obsidian compatibility.

  • Phase 2 (scaffolded, not built): hold papers in context — fetch arXiv full text, chunk, embed, and let recall pull relevant passages in. The Paper schema is ready.

Non-goals: custom graph visualization (Obsidian does it), web UI, autonomous experiment execution, raw metric-curve storage (reference W&B/MLflow), auto-committing to git.

License

MIT

Available Tools

17 tools
add_paperC

Add an external reference (metadata + key claims). Cite it from hypotheses to ground your rationale.

Inputs: title (required), id, key_claims, topic_tags, +1 more. Returns: object

calibrationA
Read-onlyIdempotent

Prediction-accuracy report: the fraction of experiments where predicted direction == observed direction. Answers 'where is my intuition miscalibrated?'. Scope by diagnostic, tag, or intervention.

Inputs: tag, diagnostic_id, intervention_id. Returns: object

causal_mapA
Read-onlyIdempotent

Aggregated intervention → Δdiagnostic subgraph across all experiments in scope: what has ever moved a diagnostic, in which direction, how often, by how much. A mini meta-analysis of your own runs. Scope by problem_id or tag.

Inputs: tag, problem_id, emit_canvas. Returns: object

close_experimentA
Idempotent

Finalize: compute observed_effects, prediction_match (per-diagnostic + overall) and the verdict (supports/refutes/inconclusive), then propagate the status to the hypothesis. The verdict is COMPUTED from the primary prediction vs the observation — you cannot assert 'confirmed'. On a positive match it RETURNS a suggestion to commit your git_ref; it never commits for you.

Inputs: experiment_id (required), notes. Returns: object

create_problemA

Register an open research question / failure mode. Do this once per distinct problem. Use recall first to check it isn't already logged.

Inputs: title (required), id, paper_refs, topic_tags, +2 more. Returns: object

get_problemA
Read-onlyIdempotent

Fetch a single problem by id.

Inputs: id (required). Returns: object

list_diagnosticsB
Read-onlyIdempotent

List the registered diagnostic basis.

Inputs: no named parameters. Returns: object

list_interventionsA
Read-onlyIdempotent

List registered interventions.

Inputs: no named parameters. Returns: object

list_problemsA
Read-onlyIdempotent

List problems, optionally filtered by status {open|resolved|parked} or tags.

Inputs: tags, status. Returns: object

propose_hypothesisA

Propose an explanation/intervention for a problem. You MUST commit ≥1 predicted_effect on a REGISTERED diagnostic BEFORE testing — this is rejected otherwise. Committing the prediction up front is what turns this log into a calibration signal. Call recall first. To revise a dead end, pass supersedes rather than deleting it.

Inputs: statement (required), problem_id (required), predicted_effects (required), id, +7 more. Returns: object

recallA
Read-onlyIdempotent

CALL THIS FIRST, before proposing any hypothesis (pre-mortem gate).

Returns ranked past hypotheses/experiments. Refuted and inconclusive matches are ranked to the TOP with a one-line reason they failed and the diagnostic delta that killed them — so you never silently re-walk a dead end. Also returns a calibration note if the query touches a diagnostic with a track record. After reading this, propose a NEW hypothesis (or supersede a prior one) with a committed prediction.

Inputs: query, problem_id, topic_tags, problem_tags. Returns: object

record_diagnosticsA
Idempotent

Record before/after values for measured diagnostics. Re-recording the same diagnostic overwrites it (idempotent). Observed deltas/directions are derived.

Inputs: measurements (required), experiment_id (required). Returns: object

register_diagnosticA

Register a named, fixed, measurable metric. Diagnostics are a DELIBERATE fixed basis — comparability across experiments is what makes the causal map possible. Do this before predicting on a metric.

Inputs: name (required), id, unit, direction, +1 more. Returns: object

register_interventionC

Register a named, reusable change (e.g. unfreeze-projector). Reused across hypotheses so the causal map can attribute deltas to it.

Inputs: name (required), id, topic_tags, description. Returns: object

reindexA
Idempotent

Rebuild the disposable SQLite index from the markdown vault (files are truth). Safe to call anytime; produces identical query results.

Inputs: no named parameters. Returns: object

start_experimentA

Begin testing a hypothesis. If git_ref is omitted the server reads the current branch@commit from git HEAD. Reference the external W&B/MLflow run — this server stores verdicts, not loss curves.

Inputs: hypothesis_id (required), id, git_ref, config_note, +1 more. Returns: object

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.7/5.0

    Scored across 17 tools

    Disambiguation5/5

    Each tool has a distinct purpose: registration, listing, core experimental workflow, paper linking, and analysis. No two tools appear to overlap; even 'recall' is uniquely positioned as a pre-mortem gate.

    Naming Consistency3/5

    Most tools follow a verb_noun snake_case pattern (register_diagnostic, list_problems, propose_hypothesis). However, a few deviate: 'recall', 'causal_map', 'calibration', and 'reindex' use single verbs or nouns, creating minor inconsistency.

    Tool Count5/5

    With 17 tools, the set is well-scoped for a research experiment tracking system. Each tool serves a necessary role in registration, experimentation, analysis, or maintenance, without feeling bloated or sparse.

    Completeness4/5

    The tool set covers the full lifecycle (register, recall, problem definition, hypothesis proposal, experiment execution, diagnostics recording, closure, and analysis). Minor gaps include no explicit delete or update tools, but the core workflow is complete.

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      D
      maintenance
      An MCP server that enables AI agents to observe and interact with trackio experiment tracking, providing tools for managing ML experiments through natural language.
      3
      MIT
    • A
      license
      Not graded
      quality
      D
      maintenance
      An MCP server enabling AI agents to record, query, and share structured problem-solving experiences with human review and confidence decay.
      9 npm
      MIT