# Open Knowledge Fabric (OKF)

A personal knowledge base that an AI assistant reads and writes through one MCP
server. The notes are plain markdown files with YAML frontmatter, in a git repo
you own. Between the assistant and those files sits the **Librarian**: ordinary,
deterministic Python with no LLM in it. It assigns ids, files notes, validates
every write against a schema, enforces each note's lifecycle, and searches.

The assistant does the judgment: what to write, what a new note makes false,
how to rerank what search returns. The Librarian does the bookkeeping and refuses
bad writes. Search runs on your machine: word scoring plus a small local
embedding model (`bge-small`), so your text never leaves the box.

Built by Sumit for his own notes, and run daily against several hundred of them.
This repo has the code and a fictional example corpus; no real notes are included.

## The note design: three layers

Every note (a *node*) has three layers, read in this order:

1. **Metadata.** Title, type, project, tags, and `asks`: up to eight questions a
   stranger would type when this note is the answer. Search indexes `asks` like
   the summary, so a note can be found by someone who has forgotten its words.
2. **Summary: the current state.** Up to eight `- ` bullets. Bullet 1 says what
   the note is. Each other bullet states one point the note *currently* makes and
   ends with `→ § <heading>`, the section of the body that holds the detail. When
   a point changes, its bullet says so: `(changed 2026-09-02: was 2pm)`. The summary
   is rewritten whenever the note changes.
3. **Body: the long form.** It is never overwritten. Corrections are appended as
   dated blocks, and a body replacement that would drop existing text is refused
   unless the caller passes an explicit flag. The body is the history; the summary
   is the answer of record.

```markdown
---
id: okf_20260902_064500_d002
type: decision
title: Wholesale orders close at noon the day before
summary: |-
  - The current wholesale cut-off: cafés order by noon for the next morning's delivery.
  - Noon gives the evening shift the full mix before it starts → § Why
  - Replaces the 2pm cut-off (changed 2026-09-02: was 2pm) → § Why
status: active
maturity: established
canonical: true
project: bakery
asks:
- what time do cafés have to order by
- can a café order after lunch
supersedes: okf_20260714_070000_d001
---
# Wholesale orders close at noon the day before
## Why
…
```

Replacing a note does not delete it. The new note `supersedes` the old one; the
Librarian marks the old note `superseded` and links both ways. Search then leaves
superseded notes out and tells you history exists.

## Quickstart (5 minutes, over stdio)

You need Python 3.11+ and git.

```bash
git clone https://github.com/sumit-anfy/open-knowledge-fabric.git
cd open-knowledge-fabric
python3.11 -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/python -m librarian.validate examples/corpus     # OK — 0 error(s)
```

The server runs against the bundled example corpus by default: 37 notes from a
fictional baker, Rowan, covering every note type. To keep the repo clean while
you experiment, work on a copy:

```bash
cp -r examples/corpus ~/okf-demo
```

**Claude Code:**

```bash
claude mcp add --transport stdio --env OKF_ROOT=$HOME/okf-demo okf -- "$PWD/.venv/bin/python" "$PWD/server.py"
```

**Claude Desktop:** add this to `claude_desktop_config.json` (Settings →
Developer → Edit Config), with absolute paths, then restart Claude Desktop:

```json
{
  "mcpServers": {
    "okf": {
      "command": "/absolute/path/to/open-knowledge-fabric/.venv/bin/python",
      "args": ["/absolute/path/to/open-knowledge-fabric/server.py"],
      "env": { "OKF_ROOT": "/Users/you/okf-demo" }
    }
  }
}
```

Then ask Claude things like:

- *"Brief me: what time do the cafés have to order by, and why did it change?"*
- *"Who owns the bakery?"*
- *"Create a note in garden: the beans failed because they went in too early."*
- *"Run the corpus audit."*

The first search downloads the embedding model (about 64 MB) into a local cache.
Until it is ready, and wherever `fastembed` is not installed, search falls back
to words only and says so (`semantic: false`).

To start your own corpus, copy `examples/corpus/schemas/` into an empty folder,
edit `PROJECT_REGISTRY` in `schemas/type-registry.md` to list your own homes,
`git init` it, and point `OKF_ROOT` at it. Set `OKF_GIT_COMMIT=1` to have every
write committed. All settings are in [`.env.example`](.env.example).

Run the tests (about 30 seconds). The write tests run on throwaway copies of the
example corpus:

```bash
.venv/bin/pip install -r requirements-dev.txt
.venv/bin/python -m pytest
```

## The tools

| Tool | What it does |
|---|---|
| `brief` | Search. Returns the best-matching notes with bodies, their neighbours, and what was left out as stale |
| `read_node` / `list_nodes` | Read one note by id; list by type and status |
| `create_node` / `update_node` | The write path: validated, filed, state machine enforced |
| `validate_corpus` | Check every note against the schema |
| `audit_corpus` | Weekly triage: orphans, stale links, near-duplicates, suggested links |
| `resolve_suggestion` | Accept or dismiss a suggested link |
| `propose_deletion` / `delete_node` | Delete only retired, unlinked notes; hand anything else to a human |

A session note must end with a close-out (Keep / Conflicts / Your call /
Tasks), and every ruling it lists must cite the note it is filed in. The server
refuses a session note that skips either.

## Architecture

```
Claude (Desktop, Code, claude.ai)
   │  MCP: stdio, or HTTP behind TLS with a bearer token or GitHub OAuth
server.py           thin FastMCP wrapper: ten tools, auth, no logic
   │
librarian/          the deterministic core (no LLM, no network)
   validate.py      schema + registry, read live from the corpus's own schema notes
   store.py         the write path: lock, validate as a set, atomic replace,
                    state machine, supersession, body guard, git commit/push
   context.py       brief(): hybrid ranking, intent-aware type priors, aliases
   embed.py         local bge-small embeddings (fastembed, ONNX), cached in ~/.okf
   sections.py      long notes searched section by section
   summary.py       the bullet-summary format and its checks
   cluster.py       after a write: "which neighbours does this make false?"
   redundancy.py    after a rule is written: "does a canonical note already say this?"
   completeness.py  session rulings must cite the note they are filed in
   audit.py         weekly triage, link suggestions, purge candidates
   findability.py   weekly run of held-out stranger questions, logged as a note
   │
your corpus         markdown + YAML frontmatter, its own git repo
   schemas/         node.schema.md and type-registry.md are themselves notes
okf_web/            Starlette graph explorer and markdown editor, password login
deploy/             systemd units, Caddy, a pull-and-test auto-deploy timer
```

Some design choices:

- **The vocabulary lives in the corpus.** Note types, folders and project homes
  are YAML blocks inside `schemas/type-registry.md`. The validator reads them
  live; changing your homes is an edit to a note, not to the code.
- **Fail closed.** No registry means no writes. HTTP without auth refuses to
  start. A check that cannot run reports a failure, not a pass.
- **Search only adds.** The embedding layer and the aliases can reorder results
  or add matches, but never remove a word match. Every score is broken down in
  the output (`score_lexical`, `score_semantic`, `prior`), so you can see why a
  note ranked where it did.
- **Live data stays in its own app.** Tasks, spreadsheets and calendars are
  `resource` notes that point at them, never copies that go stale.

## Retrieval numbers (example corpus only)

`python tests/retrieval_report.py` prints these. They are measured on the
37-note example corpus with the benchmark in `tests/retrieval_benchmark.yaml`
(48 questions) and the held-out bank in `tests/findability_seed.yaml` (20 notes,
two questions each, written without reusing the note's own words).

| | Words + embeddings | Words only |
|---|---|---|
| Benchmark top-1 | 46/48 | 44/48 |
| Benchmark top-5 | 47/48 | 48/48 |
| Held-out questions, top-1 | 15/20 | 5/20 |
| Held-out questions, in the default pack of 8 | 20/20 | 14/20 |
| Held-out short phrases, in the default pack of 8 | 20/20 | 16/20 |

Read these as a smoke test, not a quality claim: the same person wrote the corpus
and the questions, and 37 notes is an easy haystack. The useful signal is the
gap between the two columns on held-out wording. That gap is what the embedding
layer is for.

## What is not built

- **No LLM inside.** The server never calls a model. Summaries, `asks`, and the
  "what does this make false?" review are written by the assistant that calls
  the tools.
- **No importers.** Nothing reads Notion, Obsidian or Apple Notes.
- **Single owner.** One corpus, one owner. Auth decides who gets in, with no
  per-note permissions, no sharing, and no multi-user editing.
- **No packaging.** Run from a checkout; there is no `pip install okf` and no
  Docker image.
- **No encryption at rest.** Notes are plain files. Protect the disk and the repo.
- **Edges are only links.** `relates_to` links are plain lists. The typed-edge
  vocabulary in the schema (supports, contradicts, …) is not used yet.
- **Tested on macOS and Linux only.** It has not been run on Windows.
- **Scheduled work is external.** The weekly audit is meant to be run by a
  scheduled assistant task calling `audit_corpus`; this repo ships no scheduler
  beyond the findability run inside the HTTP service.

## Licence

[MIT](LICENSE).