# WikiBrain

**Personal knowledge base.** A self-hostable implementation of Andrej Karpathy's **LLM Wiki** pattern: you curate sources, an AI agent compiles them into a persistent, interlinked Markdown wiki, and every AI client you already use (Cursor, Claude, ChatGPT, Claude Code…) reads and writes the same wiki through MCP.

> **The hosted service at wikibrain.app was shut down on 2026-11-09.** The code stays here under AGPL-3.0 and is fully usable: run your own instance with `docker compose` (see [DEPLOY.md](DEPLOY.md)). If you had an account on the hosted service and need help with your data, write to hello@wikibrain.app.

## The idea: Karpathy's LLM Wiki

This project is a direct implementation of the pattern Andrej Karpathy described in his note [**LLM Wiki**](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f) (2026). In his words, the LLM "incrementally builds and maintains a persistent wiki" that sits between you and your raw sources, rather than re-reading the sources on every question as RAG does — so knowledge is compiled once and compounds.

WikiBrain keeps his design as-is and only adds a server and plumbing:

| Karpathy's note | In WikiBrain |
|---|---|
| Three layers: `raw/` sources, `wiki/` compiled pages, a schema the LLM reads first | The same three folders; `schema/` is served to agents through the `get_instructions` tool |
| `index.md` as the catalog, `log.md` as the append-only journal | Created by every template; the agent updates both on each ingest |
| Three operations: Ingest, Query, Lint | The same three, runnable from Cursor / Claude via MCP or from the web with your own API key |
| Obsidian + Claude Code on one machine | One server for all your clients (MCP with tokens or OAuth 2.1), versioned, with import, bibliography and Zotero on top |

The built-in help page (`/help/karpathy` on your instance) carries a longer summary of the note; the gist itself is the canonical reference.

## What it does

- **Three layers.** `raw/` holds immutable sources, `wiki/` holds AI-compiled pages, `schema/` holds the compilation rules the agent reads before writing. `index.md` and `log.md` follow Karpathy's conventions.
- **Ingest, Query, Lint.** Sources that no wiki page links to are listed as *pending*; the agent ingests them (in Cursor via MCP, or on the web with your own API key), answers questions with citations in a chat panel, and health-checks the wiki (orphans, broken links, contradictions) into `wiki/lint/`.
- **Bring sources from anywhere.** URLs (readability + metadata, DOI → Crossref, arXiv, PubMed, headless fallback for SPAs), PDF, Word, HTML, Markdown, pasted text, images. BibTeX / CSL-JSON import and export. Zotero sync.
- **For researchers.** Bibliographic front-matter (`doi`, `authors`, `year`, `venue`, `citation_key`), pandoc-style `[@citekey]` citations rendered as (Author, Year) with an automatic reference list, a literature table view, BibTeX export that works with pandoc.
- **Share, script, install.** Any page gets a read-only public link (`/s/<token>`, revocable). The same MCP token works as an API key on the REST API (`Authorization: Bearer`) for scripts, cron jobs and Claude Code hooks. The web app is an installable PWA with an offline shell.
- **MCP server built in.** Six tools (`get_instructions`, `search_notes`, `read_note`, `create_note`, `update_note`, `list_folder`) over Streamable HTTP with optimistic locking. Bearer tokens for Cursor; OAuth 2.1 with dynamic client registration for Claude.ai, ChatGPT and other connectors.
- **Web UI.** Three-pane editor with live preview, backlinks, version history and rollback, force-directed graph with a timeline, front-matter table view (Dataview-style), Mermaid diagrams, Marp slide pages, KaTeX. Traditional Chinese and English interface; write your wiki in any language.
- **No lock-in.** One-click export of the whole wiki as an Obsidian-compatible Markdown zip, plus `.bib` / CSL-JSON.

## Architecture

One Node service and one PostgreSQL database. Every AI client talks to the same wiki through MCP; the web UI and the server-side agents use the same six tools.

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="docs/diagrams/architecture-dark.svg">
  <img alt="WikiBrain system architecture: clients (browser, Cursor/Claude Code, Claude.ai/ChatGPT, scripts) → edge (security headers, rate limits, identity) → one Node service (MCP server, web API, OAuth 2.1 server) → domain modules (notes, agent runner, import/bibliography/Zotero, outbound guard) → PostgreSQL, model providers, external bibliographic sources" src="docs/diagrams/architecture-light.svg" width="960">
</picture>

How knowledge moves between the three layers — Karpathy's Ingest / Query / Lint, all done with the same six tools, by your AI client over MCP or by the server-side agent with your key:

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="docs/diagrams/knowledge-flow-dark.svg">
  <img alt="Knowledge flow: schema/ rules are read first; raw/ sources stay pending until Ingest writes wiki/ pages and updates index.md and log.md; Query reads the index then pages and can save cited answers to wiki/queries/; Lint scans the wiki and reports orphans, broken links and contradictions to wiki/lint/" src="docs/diagrams/knowledge-flow-light.svg" width="960">
</picture>

The diagrams are generated by `scripts/render-diagrams.py` (no dependencies); edit the data there and re-run.

## Self-host

```bash
git clone <this repo> && cd wikibrain
export BETTER_AUTH_SECRET=$(openssl rand -hex 32)
mkdir -p secrets && openssl rand -hex 32 > secrets/wikibrain_key   # encrypts users' API keys; kept out of env vars
export APP_URL=http://localhost:3000        # your public https URL in production (e.g. https://wiki.example.com)
docker compose up -d
open http://localhost:3000
```

Verification emails are printed to the container log unless `RESEND_API_KEY` is set. Google sign-in is enabled when `GOOGLE_CLIENT_ID`/`GOOGLE_CLIENT_SECRET` are set. The image includes headless Chromium so JavaScript-rendered pages can be imported (about 1.8 GB); set `IMPORT_HEADLESS=0` to disable it.

See [DEPLOY.md](DEPLOY.md) for self-hosting with Docker behind a reverse proxy, and `.env.example` for every variable.

## Connect an AI client

- **Cursor / Claude Code:** Settings → *Connect Cursor* generates a token and the `mcp.json` snippet (`{ "url": "https://<host>/mcp", "headers": { "Authorization": "Bearer …" } }`).
- **Claude.ai / ChatGPT / any OAuth-capable client:** add a custom connector with the URL `https://<host>/mcp`, sign in, press *Allow*. Connections are listed in Settings and can be revoked.
- **Web auto-ingest and chat:** store your own Anthropic, OpenAI or OpenRouter key in Settings; the server runs the agent with it. Token and cost statistics are shown per job.

## Develop

```bash
cp .env.example .env         # DATABASE_URL, BETTER_AUTH_SECRET, DEV_MCP_TOKEN
createdb wikibrain           # local PostgreSQL 16
npm install && npm run db:seed
npm run dev                  # API on :3000, web on :5173
npm test                     # backend end-to-end tests (real Postgres)
python3 test/e2e/m3_web.py   # Playwright E2E against the dev servers
```

Stack: Node 24, TypeScript (strict), Express 5, PostgreSQL (plain SQL migrations), `@modelcontextprotocol/sdk`, better-auth, React 19, Vite, Tailwind 4. `docs/` holds technical notes such as the URL import test matrix and the original UI prototype.

## License

AGPL-3.0. You may self-host and modify freely; if you offer a modified version as a network service you must publish your changes. The former hosted service's operator console and backup jobs are not part of this repository.