EpsilonGraph

A typed knowledge graph + epistemic ledger over an Obsidian vault, exposed to coding agents (Claude Code and anything else that speaks MCP).

It answers questions that no RAG system answers:

• What grounds the rule scaffold split is the default — and has it been tested, in how many projects?

• If I change this note, which other notes become wrong (not merely stale)?

• What does the vault itself admit it does not know?

• Give me the 8k-token subset of this 160k-token vault that this task needs.

[!note] The mathematics is written down Read THEORY.md first if you want to know why any of this should be believed. It states what each sub-system is mathematically, and then says which claims are proved, which are measured, which are mine, and which are still guesses.

The short version: blast radius is a semiring-provenance transitive closure (Green et al., PODS 2007), context packing is submodular maximization under a knapsack constraint (Sviridenko 2004; Amanatidis et al., TCS 1060, 2026), and RRF's k has a derived bound where the practitioner literature has only folklore. Writing those three down found four real bugs that reading the code did not.

Correction, 2026-10-12. A literature check found that the packing formulation is already published — arXiv:2607.00725 (Jul 2026) casts reader-context construction as budgeted monotone submodular maximization, with the task-level eval this repo lacks. That claim is withdrawn. The replacement candidate is a dependency-closure constraint under a budget, which is measurably outside that literature because closure terms are supermodular. THEORY.md §3 and §5. What replaced it, per prior work, is the related-work table below — one row per work, each naming the quantity it leaves open.

[!note] The product view Evaluating this as a product rather than a research project — install, resource auto-acquisition, multi-vault, distribution, scale — is written up in PRODUCT.md.

Related work: the scope map

What each prior work models, what it leaves open, and whether this project's measured results contradict it, confirm it, or are orthogonal (touch a different quantity). Every cell is drawn from RESEARCH.md §4 and §P8.1 and THEORY.md §5; no citation or number appears here that is not already there.

The three quantities this project stands on — h({i}) / the feasibility frontier, the supermodularity of the closure terms, and the present/correct/given it decomposition — are each attributed above to exactly one prior work that lacks them (RESEARCH.md §P8.1).

Related MCP server: Brainstem

Why this is not a RAG project

Measured on the live vault (/home/hugo/Documents/Obsidian, 103 notes):

Retrieval collapse is documented at the 10K–100K document range. At 103 notes this is a long-context selection-and-ordering problem, not a recall problem. The gap the vault actually has is epistemic state — grounding, evidence, cross-project status, contradictions, admitted gaps, blast radius — and the vault already models all of it. epsilongraph supplies the missing interface.

See DESIGN.md for the full argument, the boundary conditions that exclude GraphRAG/RAPTOR/semantic chunking at this size, and the explicit not-doing list.

Install

No third-party dependencies. Python 3.11+ (stdlib sqlite3 with FTS5).

cd ~/codes/EpsilonGraph
python3 -m venv .venv && source .venv/bin/activate   # a venv: Arch's PEP 668 blocks system pip
pip install -e .            # gives you the `epsilongraph` console script
# or, without installing anything:
python3 -m epsilongraph stats

Then one command fetches and verifies every external resource:

epsilongraph acquire status          # what is missing, and the exact fix for each
epsilongraph acquire obsidian-mcp    # checksummed prebuilt Layer 0 binary
epsilongraph acquire model           # ollama server + qwen3.5:4b (pull)

Optional adapters for URL/file acquisition (installed into the venv, never required):

pip install -e '.[acquire]'          # trafilatura, markitdown, html2text
# or pin-by-pin:
.venv/bin/epsilongraph acquire repo install trafilatura

Vault resolution order: --vault → $EPSILONGRAPH_VAULT → ~/.config/epsilongraph/config.json → Obsidian's own registry (obsidian.json, the vault marked "open": true) → cwd. The index lives at ~/.cache/epsilongraph/index.sqlite — outside the vault, so it is never committed and never syncs. The Layer 0 binary, the OSS clones, fetched content, and the acquisition ledger live under ~/.cache/epsilongraph/ too.

Installed state (2026-10-11)

Both MCP servers are registered in ~/.claude.json at user scope and both report ✓ Connected:

epsilongraph:       python3 -m epsilongraph.mcp_server                                    ✓
obsidian-mcp: /home/hugo/codes/EpsilonGraph/vendor/obsidian-mcp/target/release/...  ✓

The skill is installed at ~/.claude/skills/vault-retrieval/. The config was backed up to ~/.claude.json.bak-kgraph-<timestamp> before the change, and the three pre-existing servers (github, filesystem, MiniMax) are untouched.

obsidian-mcp points at the local release build rather than cargo install, so vendor/ must stay. If you would rather not depend on vendor/, run cargo install obsidian-mcp (~31 min) and change the command to obsidian-mcp.

LICENSE: MIT — added 2026-10-16 when the repo went public.

Use

epsilongraph index                       # build; ~0.05s for 103 notes
epsilongraph stats                       # layer / provenance breakdown
epsilongraph skeleton                    # the ~29k-token decision layer (L1)
epsilongraph pack "how should I lay out a new training project" --budget 8000
epsilongraph claim "scaffold split"      # grounding + evidence + projects + contradictions
epsilongraph impact "Principles/Planning Before Execution"   # blast radius
epsilongraph gaps                        # what the vault admits it does not know
epsilongraph search "docking power"
epsilongraph check                       # staleness + integrity
epsilongraph acquire status              # every external resource, with its fix
epsilongraph graph report                # derived-graph connectivity

Progressive disclosure ladder

pack runs three passes over the budget: cores first, then key points and gaps, then graph context. A tight budget drops detail, never candidates — and the output states what it dropped.

Derived edges (--auto)

Every index now carries a second, derived graph next to the curated one (graph.py): wikilink / backlink from the author's own links, cotag, couples (co-citation), and lexical similarity. Two rules keep it honest:

• derived edges live in their own table and are never consulted by impact / load-bearing — a guessed edge must not be able to falsify a note;

• retrieval uses them only when asked: search --auto, pack --auto, neighbors --auto, ranked below every lexical result (auto:* is glossed with its kind and weight so a reader can tell inference from curation).

epsilongraph graph build                 # rebuild the live index + derived graph
epsilongraph graph report                # connectivity: components, orphans, kinds
epsilongraph search "packing" --auto     # derived context, opt-in
epsilongraph neighbors "Flow Matching" --auto

graph build --source <any markdown tree> builds a graph over foreign markdown (an acquired repo, a fetched inbox) in its own index under ~/.cache/epsilongraph/graphs/ — the live vault's index is never touched.

Missing-link suggestions (suggest at most, never auto-write)

graph suggest ranks unlinked note pairs by derived evidence (co-citation first, lexical last), asks the local reader model for one of the eight typed buckets, and writes a report — paste-ready lines for the note's ## Relations block. Nothing is ever written into the vault: the typed graph stays hand-curated, which is the product's core claim about quality.

epsilongraph graph suggest --top 8 --out suggestions.md

Resource auto-acquisition

Everything this project does not own — the Layer 0 Rust binary, the reader model, the adapted OSS projects — is fetched, verified, and recorded by one command. All of it stdlib-only, checksum-verified, and --dry-run-able.

epsilongraph acquire status                 # one screen: what is missing, and the fix
epsilongraph acquire obsidian-mcp           # checksummed prebuilt binary -> ~/.cache/epsilongraph/bin/
epsilongraph acquire model                  # ollama + qwen3.5:4b (starts the server, pulls the model)
epsilongraph acquire repo trafilatura       # pinned-commit clone -> ~/.cache/epsilongraph/repos/
epsilongraph acquire fetch --url https://example.com/doc --to ./inbox
epsilongraph acquire import --path paper.pdf --to ./inbox

• the obsidian-mcp download verifies the release's own sha256 digest before the binary is made executable, and deletes on mismatch;

• repos are pinned by immutable commit SHA and hash-locked in ~/.cache/epsilongraph/acquisitions.json — a later fetch that does not match the recorded hash is reported as tampering;

• fetch prefers trafilatura, import prefers markitdown; when neither is installed the stdlib fallback runs and says which path it took;

• fetched notes carry acquired: true provenance so the vault's epistemic ledger never mistakes external content for ours.

MCP server

Dependency-free newline-delimited JSON-RPC 2.0 over stdio. Nine tools:

vault_pack vault_claim vault_impact vault_gaps vault_search vault_neighbors vault_get vault_skeleton vault_stats

python3 -m epsilongraph.mcp_server --selftest   # drives a real subprocess session
python3 -m epsilongraph.mcp_server              # stdio server

Merge integrations/claude/mcp.json.snippet into ~/.claude.json, and append integrations/claude/CLAUDE.md.snippet to your CLAUDE.md. The skill at integrations/claude/skills/vault-retrieval/SKILL.md installs to ~/.claude/skills/vault-retrieval/.

Eval

python3 evals/run_eval.py                                    # dev set (30)
python3 evals/run_eval.py --queries evals/queries-holdout.tsv # held-out (30)
python3 evals/tune_rrf.py                                    # choose k, check on holdout
python3 evals/compare_pack.py                                # 3 packers, paired sign test
python3 evals/compare_packers.py                             # 8 packers x 2 objectives x AiC proxy
python3 evals/closure_gap.py                                 # the §3 measurement, in one run
python3 evals/synth_corpus.py --out /tmp/epsilongraph-synth        # generate an off-vault corpus
PYTHONPATH=vendor/bm25s python3 evals/compare_bm25s.py       # second BM25, cross-check

Two 30-query sets with expected note paths, R@1/3/5 + MRR, a four-config ablation, and a separate coverage metric for graph expansion.

dev (queries.tsv — where a parameter may be chosen):

config        R@1    R@3    R@5     MRR  misses
lexical      0.70   0.97   0.97   0.821  0
+priors      0.70   0.97   0.97   0.821  0
+graph       0.70   0.97   0.97   0.821  0
full         0.70   0.97   0.97   0.821  0

held out (queries-holdout.tsv — where it is checked):

lexical      0.80   0.97   0.97   0.871  0
graph expansion:  without graph 0/30 (0%)   with graph 24/30 (80%)

The eval has falsified three of the design's own claims, each recorded in the code rather than quietly dropped:

1. Structural priors do not help. They first measured as harmful (MRR 0.817 → 0.767); after the RRF fix they measure as exactly +0.000 on both sets. Off by default.

2. Graph expansion's MRR delta is exactly 0.000 — because expansion is kept out of the ranked list by construction, so MRR cannot move. The right metric is coverage: 0/30 → 24/30.

3. RRF_K = 60 was wrong. See below.

Packing: rank order vs greedy vs Sviridenko

retrieve.pack used to fill the budget in rank order, optimising nothing in particular. packopt.py writes the objective down (THEORY.md §2) and scores all three algorithms on it. Untuned weights, budget 1800 tokens:

split     algorithm    mean f   vs rank_order   sign test
DEV       rank_order    3.098   --              --
(30 q)    greedy        3.576   +15.4%          20W/3L,  p=0.0005
          sviridenko    3.692   +19.2%          20W/0L,  p=0.0000

HOLDOUT   rank_order    3.167   --              --
(30 q)    greedy        3.766   +18.9%          27W/1L,  p=0.0000
          sviridenko    3.813   +20.4%          28W/0L,  p=0.0000

Holdout improves more than dev, and nothing was tuned. rank_order lands within 4% of best_single (3.167 vs 3.041 on holdout) — the old pack was, in objective terms, "pick the single best note and stop".

⚠️ The +20.4% above is not evidence of anything. It was measured before any task-level proxy existed. With one (answer-in-context, ported from the paper that owns this formulation), the ordering reverses: rank_order beats every optimizer on hit@pack, rule_coverage and AiC. The cause is structural — three of f's five terms do not see the query. Full table and the fix in THEORY.md §2; run python3 evals/compare_packers.py.

The dependency-closure gap

Where this is going: RESEARCH.md — the plan and the decision points. The executable version is todo.json (15 tasks, each with a falsifiable done_when, all runnable locally: pure Python + qwen3.5:4b via ollama).

The gap below is the project's research problem. Short version: 10 of 103 notes have no dependency-closed pack at all under an 1800-token budget (h({i}) runs min 17 / median 816 / max 2495 tokens), while at 4000 every one of them does — a feasibility threshold, not a binary verdict. Reframed as h(S) = cost_full(S) + cost_skel(closure(S)) it becomes a monotone submodular constraint (verified exactly), so the problem is a non-monotone submodular objective under a submodular knapsack — the corner neither half of the published literature covers.

The vault's own loading rule ("rules before procedures … you will follow the steps while violating the intent") is a constraint on what a pack must contain, and no packing objective expresses it. Measured on the 30 holdout queries: 24 packs contain the note that answers the query, and 0 of those 24 carry its dependencies.

The obvious fix is unavailable. Both natural closure terms — edges with both endpoints inside the pack, and edges leaving it — are supermodular (worst violation +2.0), so a closure-aware packer is outside the submodular-packing literature that the 2026 papers, and this repo, live in. Three mechanisms were built and all three fail: post-hoc repair reaches 100% closure at 0.00 hit@pack, closure-by-construction at 0.03, and a soft term is a no-op (the budget is already spent). Details in THEORY.md §3; run python3 evals/closure_gap.py.

Does it hold off-vault? (evals/synth_corpus.py)

Every number above comes from one corpus: the author's own vault, 103 notes, hand-typed relations. That is the project's weakest flank — a result that holds only on a graph you built by hand is a case study, and no second real vault has the typed-edge, two-tier structure the question needs. So the second corpus is generated, with the dependency graph, each note's closure size, and each query's target set by construction rather than observed.

python3 evals/synth_corpus.py --out /tmp/epsilongraph-synth --notes 120 --depth 3
python3 -m epsilongraph --vault /tmp/epsilongraph-synth index
EPSILONGRAPH_VAULT=/tmp/epsilongraph-synth EPSILONGRAPH_INDEX=/tmp/epsilongraph-synth.sqlite \
    python3 evals/closure_gap.py --queries /tmp/epsilongraph-synth/_queries.tsv

It reproduces the vault's three structural facts — the gap (28/30 packs contain the answer, 15/30 carry its dependencies), both closure terms supermodular at +2.0, f itself exactly submodular — and it turns the vault's observed slot-competition explanation into a controlled curve. --depth sets closure size; the closure-requiring packer's hit@pack falls monotonically as it rises while the unclosed packer stays flat:

closure median      2      5      6     12     <- --depth 2 3 4 6
amort+closed     1.00   0.90   0.93   0.63    hit@pack
amort+rel        1.00   1.00   1.00   0.90    hit@pack, unclosed

Closure stays affordable at every depth — 100% of notes are closable at 1800 — and still costs retrieval. That separation, "affordable" against "free", is what the live vault could show but not vary. The generator prints a manifest so the packer's output can be checked against ground truth fixed before it ran.

Scope, stated so the table is not over-read: the synthetic notes carry planted lexical signal and a uniform-depth graph, where the real vault has neither. Only the shape of the tradeoff transfers across corpora, not absolute R@k.

The reader grid was re-run on this corpus at two closure depths (seed 7, 120 notes, --depth 2 the control and --depth 6 the treatment, qwen3.5:4b, n=30; archived to evals/reader-eval-synth-{d2,d6}-2026-10-12.json). What transferred and what did not — one row per headline number, with a verdict — is the table in RESEARCH.md §P6.3. The short version: the depth-vs-hit@pack curve and the slot-competition mechanism transfer; the levels do not, and the closed-vs-unclosed present gap is invisible at depth 2 because a median closure of 2 leaves nothing to carry.

What the second BM25 found

evals/compare_bm25s.py ranks the same 103 notes with xhluca/bm25s — an independent BM25 on numpy sparse matrices, different tokenizer, different scoring constants, no field weights. Two independent implementations over one corpus is the only way to tell "the ranking is fine" from "my ranking agrees with itself".

It agreed on 64% of top-10 sets and disagreed usefully: it exposed a query where the correct note is ranked #1 by the title retriever and #18 after fusion.

The cause was arithmetic, not taste. Score is 1/(k + rank). With k=60 and a candidate depth of 50, every contribution lands in [1/110, 1/61] — under 2× spread. So appearing in a second list (2×) beat being ranked first (1.8×). Ubiquity outranked strength. k=60 is tuned for corpora of thousands of documents; at 103 notes it is a flattening filter.

k=1 was chosen on dev and confirmed on holdout, where MRR is monotonic in k:

The split earned its keep: k=8 scored best on dev and worse on holdout — it was fitted to the 30 dev queries and would have shipped as an improvement. evals/tune_rrf.py flags that case as FITTED automatically.

With n=30, one query is 0.033 MRR. Differences below ~0.03 are a single query and are not findings.

Vendored

Four upstream projects, cloned to vendor/ and adapted. None is imported by epsilongraph/ — the package stays dependency-free; they are measurement and infrastructure, not runtime dependencies.

The last two are the 2026-10-12 prior-art adaptation. Their ace_rag/ cannot run here (it needs Qwen2.5 3B/7B/14B from HuggingFace, which this machine cannot reach), so the two pieces that are reader-independent were extracted instead: the objective and the metric. vendor/ is gitignored except ace-rag/ace_rag/, which tests/test_closure.py imports for the parity check.

One vault-pollution note. obsidian-mcp keeps per-vault state at {vault}/.obsidian-mcp. Setting OBSIDIAN_MCP_DATA moves the heavy cache (BM25 index, embeddings) to ~/.cache/obsidian-mcp/vaults/{slug}/, but the server still creates a small vault-local .obsidian-mcp/ignore by design. It contains no markdown. epsilongraph/parse.py skips the directory anyway — that is insurance against the coupling, not a fix for a bug that happened.

Not vendored: AnswerDotAI/rerankers was the right second choice on paper — cross-encoder reranking is the one technique in the "non-negotiable tier" this project has not implemented, its core is dependency-free, and its FlashRank backend is ONNX rather than torch. It was rejected because its models download from HuggingFace, which this machine cannot reach (curl: (35) Recv failure: Connection reset by peer; PyPI is fine). A reranker that cannot load a model is not a reranker. The design's no-dense conclusion is unchanged either way: at 103 notes with R@5 0.97 there is no measured headroom to win back.

Tests

python3 -m unittest discover -s tests

126 tests in eleven groups:

• parity with the pre-existing validate_links.py on the live vault — note set, link count, broken-link set, and per-note relation buckets must match exactly. Two parsers for one corpus is a drift generator; the agreement is tested, not asserted.

• ranking — pins the three properties that measurement found broken: that rank outweighs list membership, that a near-exact title match reaches the top 5, and that the held-out set is still held out.

• closure — three kinds, kept separate on purpose. Parity: epsilongraph/aic.py must agree with the vendored original, and the ported ACE objective must be monotone and submodular on our corpus as published. Properties: both natural closure terms must stay supermodular — if one is ever found submodular, THEORY.md §3 collapses into §2 and the test says so. Invariants: the mechanisms return closed, within-budget sets, and the production packer still does not.

• docs consistency — the numbers the documents agree to repeat are recomputed from the repo and compared: the corpus size, the dependency-edge count, the test count, and the file layout. Five numbers were corrected in code and left stale in prose on 2026-10-12, each found by grepping by hand. This turns that into a failure. It deliberately allows a correction note to name the value it corrects — forbidding that would forbid the retraction itself, which is what the first version of the check did.

• reader harness — the four claims of evals/reader.py, testable without a model: that it is stdlib-only (the AST is parsed and the import set checked against a whitelist), that the request sends think: false (without it the model puts everything in thinking and every answer scores as wrong), that the answer is read from response, and that the cache key covers the whole request rather than the prompt alone.

• synthetic corpus — the artefact assertions of evals/synth_corpus.py, the off-vault validation. Every other test in this repo assumes the live vault; this one asserts that a generated corpus actually has typed dependency edges, a non-degenerate closure distribution, a depth knob that moves it, and query targets that exist in it. It exists because the generator failed in exactly that way twice during construction — a bullet-prefixed Relations header and a bucket name that was a sentence — and both times it produced a corpus that indexed cleanly, looked plausible, and had zero typed edges. A closure measurement against such a corpus reports that closure costs nothing, and is believed.

• reader report — the confidence intervals and paired tests of evals/reader_eval.py's report(), testable without a model. report() reads an acc dict of per-query 0/1 lists, which is exactly what the archived reader-eval-*.json holds, so the offline path (--from-json) is exercised both against a hand-built dict and against the real archive. The assertions require the Wilson [k/n] count and the paired [n=...] marker to be present, because a p-value without its denominator is the failure this project keeps recording — a test that merely checks "a p appears" would pass on the report that drops the n.

• unit — gloss retention, gap-bullet splitting, the table-escaped-pipe contract.

• acquire — platform→asset mapping for the Layer 0 download, checksum verification (mismatch ⇒ deleted, never kept), tarball extraction and ledgering, the stdlib HTML fallback, the one-screen status report, and the zero-runtime-dependency invariant (the optional trafilatura / markitdown / html2text adapters may only ever be imported inside a function).

• graph — the derived-edge computation (exact edge sets, thresholds, co-citation, orphan agreement) on a fixture vault; that curated pairs are never offered as missing-link suggestions; that a foreign --source gets its own index under graphs/ and never resolves the live vault's.

Layout

epsilongraph/
  parse.py       # frontmatter, callouts, glossed relations, gap bullets
  config.py      # vault resolution (5 steps), index location
  index.py       # SQLite + FTS5 build; provenance classification; staleness
  retrieve.py    # 4-retriever RRF, budgeted packing, skeleton
  ledger.py      # claim / impact / gaps — the epistemics
  severity.py    # blast radius as a semiring closure over a severity lattice
  packopt.py     # f as a set function; greedy / Sviridenko; closure machinery
  priorart.py    # the published objective (arXiv:2607.00725), ported
  aic.py         # answer-in-context, ported verbatim — the task-level proxy
  cli.py         # subcommands
  acquire.py     # resource auto-acquisition: Layer 0 binary, ollama model, OSS clones, URL/file fetch
  graph.py       # automatic KG construction: derived edges over any markdown tree, suggestions
  mcp_server.py  # stdio JSON-RPC, 9 tools, --selftest
evals/
  queries.tsv           # 30 dev queries
  queries-holdout.tsv   # 30 held-out queries (do not edit)
  run_eval.py           # R@k / MRR / ablation / graph coverage
  tune_rrf.py           # choose k on dev, check on holdout, flag FITTED rows
  compare_pack.py       # 3 packers, paired sign test -- where THEORY 2's p-values come from
  compare_packers.py    # 8 packers x 2 objectives x the AiC proxy (means, no test)
  stats.py              # the shared statistics: sign test, Wilson + bootstrap CIs, permutation
  closure_gap.py        # the dependency-closure measurement (§3)
  closure_sweep.py      # the depth x budget frontier (P1b)
  synth_corpus.py       # a second, generated corpus with a known dependency graph
  config_manifest.py    # one JSON: vault/index paths, budgets, SEED, reader model, deps
  reader.py             # the local reader: stdlib only, self-starting, cached
  reader_eval.py        # the task-level eval and its grader (P2)
  compare_bm25s.py      # second independent BM25 over the same corpus
  reader-eval-*.json    # the recorded runs, so a number in the docs is traceable
tests/           # parity + ranking + closure + reader + report + docs + unit + synthetic
integrations/    # Claude Code mcp.json / CLAUDE.md / skill
vendor/          # upstream clones — only ace-rag/ace_rag/ is committed
Makefile         # make repro (offline evals -> results/), make repro-reader, make test