vibescience-mcp
Planned integration to fetch arXiv paper full text, chunk, embed, and enable semantic recall during the scientific loop.
Allows referencing MLflow runs as external run stores for experiments.
Allows viewing and navigating the experiment vault (markdown files) in Obsidian with graph view, backlinks, and tag panels.
Allows referencing external W&B runs from experiments to avoid storing metric curves directly.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@vibescience-mcp Recall prior experiments related to learning rate tuning"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
vibescience-mcp
A scientific experiment log (MCP server) for an AI coding/research agent working on ML projects.
It is not a lab notebook and not Weights & Biases. It stores reasoning and causal claims, not training curves, and defines "confirmed" as agreement between a prediction committed before the test and the effect observed after, on a fixed diagnostic basis. That makes your log a queryable causal map ("which interventions move which diagnostics, in which direction, how often") plus a calibration signal on your own intuition.
Markdown files are the source of truth → open vault/ in Obsidian for graph view, backlinks and tag panels for free. The SQLite index is disposable and rebuilt from markdown at any time.
Why it's different
lab notebook | W&B / MLflow | vibescience | |
stores | free-text notes | loss curves, metrics | predictions, verdicts, causal claims |
"confirmed" means | you say so | — | predicted direction == observed direction (computed) |
negative results | buried | buried | first-class, ranked to the top of recall |
cross-experiment view | none | per-run dashboards | aggregated causal map + calibration |
Related MCP server: hive-exp
Install
cd vibescience-mcp
python -m venv .venv && . .venv/bin/activate
pip install -e . # add ".[embeddings]" for optional semantic recall, ".[dev]" for testsConfigure in Claude Code / Cursor / Desktop
The server speaks stdio. Point your MCP client at it and set the vault path:
{
"mcpServers": {
"vibescience": {
"command": "/root/vibescience-mcp/.venv/bin/python",
"args": ["-m", "vibescience_mcp.server"],
"env": {
"VIBESCIENCE_VAULT": "/root/vibescience-vault",
"VIBESCIENCE_EMBEDDINGS": "off"
}
}
}
}Or via the Claude Code CLI:
claude mcp add vibescience -e VIBESCIENCE_VAULT=/root/vibescience-vault \
-- /root/vibescience-mcp/.venv/bin/python -m vibescience_mcp.serverThe scientific loop (enforced by tool descriptions)
recall— always first (pre-mortem gate). Refuted/inconclusive matches rank to the top, each with a one-line failure reason and the diagnostic delta that killed it. Never re-walk a dead end.create_problem— frame the open question / failure mode.register_diagnostic/register_intervention— diagnostics are a fixed, comparable basis, not free text. Adding one is deliberate.propose_hypothesis— requires ≥1predicted_effecton a registered diagnostic before testing. The first is the primary prediction the verdict keys off. No prediction → rejected.start_experiment— auto-capturesbranch@shortshafrom git HEAD if you omitgit_ref. References an external W&B/MLflow run; never stores curves.record_diagnostics— before/after per diagnostic; deltas/directions derived.close_experiment— computesobserved_effects,prediction_match(per-diagnostic + overall) and theverdict, propagates status to the hypothesis, and suggests (never performs) a commit on a positive match. Record artifacts, headline numbers and dead ends inartifacts/key_numbers/do_not_retryrather than compressing them intonotes.
Verdicts. supports · refutes · inconclusive · directional_only (direction matched but a preregistered blocking gate failed) · null · crashed.
null is a powered null: the direction nominally matched, a blocking gate carrying a preregistered mde failed, and |delta| came in under that MDE. It is treated as a refutation of the lever at the committed scale — not a near-miss — and ranks with refutes. Without it, a sign-only test reports delta=+0.0003 at permutation p=0.65 as "the direction matched" and ranks it above inconclusive, i.e. the strongest evidence against a lever outranks "we don't know". Set mde on a gate to get this; omit it and the old two-level behaviour is preserved exactly.
Housekeeping:
doctor— reports records whose status no longer matches reality: answered problems stillopen, hypotheses rotting inproposed/testing, experiments left open, near-duplicate experiments started minutes apart (the fingerprint of a doublestart_experiment), runbooks queued on resolved problems, orphan and probable-duplicate tags. Advisory only; every finding names the tool that fixes it.merge_tags— fold a duplicate tag into its canonical form across the whole vault. The source survives as an alias, so historical spellings still resolve.update_problem(add_update=...)/update_runbook(note=...)— dated log entries instead of banners glued onto a description.
Then query your own work:
causal_map(problem_id | tag)— aggregatedintervention → Δdiagnosticsubgraph. Optional Obsidian.canvasoutput.calibration(diagnostic_id | tag | intervention_id)— fraction of predictions that matched observation. Where is your intuition wrong?reindex()— rebuild the SQLite index from markdown (idempotent).
A vibescience://guide resource ships the same workflow to any agent.
Storage layout
vault/
problems/ hypotheses/ experiments/
diagnostics/ interventions/ papers/
_canvas/ # generated Obsidian canvases
.index.sqlite # disposable — rebuilt from markdown by reindex()Each entity is one markdown file: YAML frontmatter for structured fields, body for prose + [[wikilinks]]. Fully Obsidian-compatible.
Env vars
var | default | meaning |
|
| vault path (source of truth) |
| unset |
|
|
|
|
VAULT_DIR and VIBESCIENCE_VAULT_DIR are accepted as aliases (with a warning
on stderr); --vault PATH overrides everything. If two of them disagree the
server exits rather than guessing. The resolved vault path is printed to
stderr at startup and embedded in the MCP instructions, so the agent can
always see which vault it is writing to.
Running several vaults
One project per vault. Register one server entry per project:
"vibescience": { "env": { "VIBESCIENCE_VAULT": "/root/vibescience-vault" } },
"vibescience-lightsem": { "env": { "VIBESCIENCE_VAULT": "/root/lightsem-vault" } }Verify isolation before trusting it — list_problems on each server must return
different records. A silently shared vault interleaves records from unrelated
projects, which corrupts the experiment log's role as the system of record.
Bundled workflow skill
skills/vibescience-mcp-workflow/SKILL.md ships the battle-tested procedure for
running the full loop: pair deep research (perplexity_research) + the
arxiv MCP with a real, verdict-computed experiment cycle, then dogfood it
into the vault. It encodes the hard-won pitfalls (research timeouts, making an
experiment able to refute, semi-synthetic precision protocols on unlabeled
prod data, the opt-in-vs-blanket deployment guardrail). Load it into any
Hermes/Claude session with skill_view(name='vibescience-mcp-workflow'), or read
it directly — it is plain markdown.
Record IDs
IDs are read by humans — in ls, in [[wikilinks]], in error messages. Auto-generated ids are capped at 48 characters on a word boundary plus a short hash of the full text, so a shared prefix can never collide; experiments get exp-NNN-<stem>, which keeps ls experiments/ usable as an overview. start_experiment is idempotent within a 5-minute window: a retried call returns the existing open experiment instead of minting a second record.
Migrating an older vault
python migrate_ids.py <vault_path> # dry run — shows the plan
python migrate_ids.py <vault_path> --apply # back the vault up firstRenames every over-long id, rewrites all cross-references (problem_id, hypothesis_id, parent_experiment_id, supersedes, paper_refs) and every wikilink, records the old id in former_ids so external references stay traceable, and writes id_migration.json as the redirect map. Store.resolve_id still finds a record by its old name. Nothing is deleted and no prose is edited — only identifiers change.
Tests
./run_tests.sh -q # 203 tests: verdict math (incl. powered nulls + weak_signal),
# prediction gate, full loop, negative-result ranking,
# causal_map, calibration, index rebuild, id scheme +
# idempotency, doctor, merge_tags, interpretation layer
# (lessons, stale negatives, plateau, portfolio,
# research_brief), markdown round-trip, MCP stdio smokeThe interpretation layer (2026-08)
Distilled from Prime Intellect's Measuring Autonomous AI Research (the gap
between models is experiment hygiene under noise, not idea quality) plus an
autopsy of a real vault: 61 experiments, 39 negative verdicts, zero recorded
lessons, 16/17 problems eternally open, 30/30 papers never linked to a
hypothesis. A log that records verdicts but never analysis is a junk pile.
interpret_experiment(id, lesson, failure_mode)— the act of analysis. Negative verdicts require afailure_mode(claim_false / mechanism_wrong / test_invalid / noise / execution_bug / recipe_conditional): a refute from a broken capture and a refute of the idea are different objects.recallsurfaces the authored lesson instead of a machine string;doctorreports closed-but-uninterpreted runs as debt.reopen_hypothesis(id, reason)— a negative result is a fact about (claim × recipe), not the claim.close_experimentsnapshots arecipe_fingerprint(interventions + provenance hashes); when the recipe drifts, a refute flips tostale_negative— flagged re-testable, history untouched, the re-test still goes throughsupersedes+ a fresh prediction.progress(problem_id)+synthesize_problem— the compiled trajectory (primary-diagnostic movement, verdict counts, plateau, interpretation debt) and the authored current-state-of-knowledge that turns a chronology into a knowledge base.Noise floors —
register_diagnostic(..., noise_sigma, noise_source= 'measured')makes the verdict engine demote a directionally-correct delta insidek*sigmatoweak_signal: a candidate to re-draw, neither a win to deploy nor a loss to discard. The top autonomous-research runs measured their own noise; the weak ones shipped draw luck.Ambition economics — hypotheses declare
ambition_class(tuning / mechanism / reframe) andgap_closure_estimateagainst the problem's declaredtarget; experiments carrygpu_hours.portfolio()shows where the compute went per class, and a plateaued problem (last 5 closed runs, nosupports) rejects furthertuningwithout anoverride_reason— the crumb trap made expensive.research_brief(problem_id)— compiles state + gap + dead ideas (with lessons) + rule-outs + the already-read paper exclusion list into a ready deep-research query, so every literature sweep is incremental by construction. The literature is the edge a sandboxed harness doesn't have.
Phasing
Phase 1 (this): schema + lifecycle +
recall+causal_map+calibrationover markdown, SQLite index, Obsidian compatibility.Phase 2 (scaffolded, not built): hold papers in context — fetch arXiv full text, chunk, embed, and let
recallpull relevant passages in. ThePaperschema is ready.
Non-goals: custom graph visualization (Obsidian does it), web UI, autonomous experiment execution, raw metric-curve storage (reference W&B/MLflow), auto-committing to git.
License
MIT
Available Tools
17 toolsadd_paperC
Add an external reference (metadata + key claims). Cite it from hypotheses to ground your rationale.
Inputs: title (required), id, key_claims, topic_tags, +1 more. Returns: object
calibrationARead-onlyIdempotent
Prediction-accuracy report: the fraction of experiments where predicted direction == observed direction. Answers 'where is my intuition miscalibrated?'. Scope by diagnostic, tag, or intervention.
Inputs: tag, diagnostic_id, intervention_id. Returns: object
causal_mapARead-onlyIdempotent
Aggregated intervention → Δdiagnostic subgraph across all experiments in scope: what has ever moved a diagnostic, in which direction, how often, by how much. A mini meta-analysis of your own runs. Scope by problem_id or tag.
Inputs: tag, problem_id, emit_canvas. Returns: object
close_experimentAIdempotent
Finalize: compute observed_effects, prediction_match (per-diagnostic + overall) and the verdict (supports/refutes/inconclusive), then propagate the status to the hypothesis. The verdict is COMPUTED from the primary prediction vs the observation — you cannot assert 'confirmed'. On a positive match it RETURNS a suggestion to commit your git_ref; it never commits for you.
Inputs: experiment_id (required), notes. Returns: object
create_problemA
Register an open research question / failure mode. Do this once per
distinct problem. Use recall first to check it isn't already logged.
Inputs: title (required), id, paper_refs, topic_tags, +2 more. Returns: object
get_problemARead-onlyIdempotent
Fetch a single problem by id.
Inputs: id (required). Returns: object
link_paperBIdempotent
Link a paper to a problem or hypothesis (adds a cites edge).
Inputs: paper_id (required), entity_id (required). Returns: object
list_diagnosticsBRead-onlyIdempotent
List the registered diagnostic basis.
Inputs: no named parameters. Returns: object
list_interventionsARead-onlyIdempotent
List registered interventions.
Inputs: no named parameters. Returns: object
list_problemsARead-onlyIdempotent
List problems, optionally filtered by status {open|resolved|parked} or tags.
Inputs: tags, status. Returns: object
propose_hypothesisA
Propose an explanation/intervention for a problem. You MUST commit ≥1
predicted_effect on a REGISTERED diagnostic BEFORE testing — this is
rejected otherwise. Committing the prediction up front is what turns this log
into a calibration signal. Call recall first. To revise a dead end, pass
supersedes rather than deleting it.
Inputs: statement (required), problem_id (required), predicted_effects (required), id, +7 more. Returns: object
recallARead-onlyIdempotent
CALL THIS FIRST, before proposing any hypothesis (pre-mortem gate).
Returns ranked past hypotheses/experiments. Refuted and inconclusive matches are ranked to the TOP with a one-line reason they failed and the diagnostic delta that killed them — so you never silently re-walk a dead end. Also returns a calibration note if the query touches a diagnostic with a track record. After reading this, propose a NEW hypothesis (or supersede a prior one) with a committed prediction.
Inputs: query, problem_id, topic_tags, problem_tags. Returns: object
record_diagnosticsAIdempotent
Record before/after values for measured diagnostics. Re-recording the same diagnostic overwrites it (idempotent). Observed deltas/directions are derived.
Inputs: measurements (required), experiment_id (required). Returns: object
register_diagnosticA
Register a named, fixed, measurable metric. Diagnostics are a DELIBERATE fixed basis — comparability across experiments is what makes the causal map possible. Do this before predicting on a metric.
Inputs: name (required), id, unit, direction, +1 more. Returns: object
register_interventionC
Register a named, reusable change (e.g. unfreeze-projector). Reused across hypotheses so the causal map can attribute deltas to it.
Inputs: name (required), id, topic_tags, description. Returns: object
reindexAIdempotent
Rebuild the disposable SQLite index from the markdown vault (files are truth). Safe to call anytime; produces identical query results.
Inputs: no named parameters. Returns: object
start_experimentA
Begin testing a hypothesis. If git_ref is omitted the server reads the
current branch@commit from git HEAD. Reference the external W&B/MLflow run —
this server stores verdicts, not loss curves.
Inputs: hypothesis_id (required), id, git_ref, config_note, +1 more. Returns: object
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 17 tools
Each tool has a distinct purpose: registration, listing, core experimental workflow, paper linking, and analysis. No two tools appear to overlap; even 'recall' is uniquely positioned as a pre-mortem gate.
Most tools follow a verb_noun snake_case pattern (register_diagnostic, list_problems, propose_hypothesis). However, a few deviate: 'recall', 'causal_map', 'calibration', and 'reindex' use single verbs or nouns, creating minor inconsistency.
With 17 tools, the set is well-scoped for a research experiment tracking system. Each tool serves a necessary role in registration, experimentation, analysis, or maintenance, without feeling bloated or sparse.
The tool set covers the full lifecycle (register, recall, problem definition, hypothesis proposal, experiment execution, diagnostics recording, closure, and analysis). Minor gaps include no explicit delete or update tools, but the core workflow is complete.
Related MCP Connectors
MCP server for building and testing AI agents with multi-model experimentation and insights.
Persistent, inspectable memory for AI agents with lineage, correction, and a hosted MCP endpoint.
Provenance-carrying memory for AI agents: corrections supersede, never overwrite. Free, API + MCP.
MCP server for the Fail Modes taxonomy — a knowledge base of AI system failure modes
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceAn MCP server that enables AI agents to observe and interact with trackio experiment tracking, providing tools for managing ML experiments through natural language.3MIT
- AlicenseNot gradedqualityDmaintenanceAn MCP server enabling AI agents to record, query, and share structured problem-solving experiences with human review and confidence decay.9 npmMIT
- AlicenseNot gradedqualityCmaintenanceAn MCP server that enforces a scientific-method loop for AI-driven machine learning experiments, with hypothesis gating, diagnostics, and data forensics.Apache 2.0
- AlicenseNot gradedqualityBmaintenanceA causally-ordered, rewindable event-ledger for autonomous AI agents, enabling tamper-evident audit, replay, and rollback of agent actions via an MCP server.4MIT