# wiki-compiler

A CLI + MCP server that **compiles** web sources into a structured, growing knowledge base — Obsidian-compatible, local-first, schema-driven.

> "There is room here for an incredible new product instead of a hacky collection of scripts."
> — Andrej Karpathy

Most tools give your AI *access* to your notes. This one gives your AI a *compiled knowledge structure* — backlinked wiki pages built from raw sources, organized by a schema you define.

---

## How it works

```
wiki add <url>
    │
    ▼
fetch page → strip HTML → send to Claude
    │
    ▼
Claude extracts: title, summary, key_concepts, categories, tags
    │
    ▼
vault/agent/{slug}.md  ← Obsidian-compatible markdown
    │
    ▼
wiki query "what do I know about X?"
    │
    ▼
Claude reads all vault pages → synthesized answer
```

**Dual vault architecture** (kepano's contamination insight):
- `vault/agent/` — compiler output, messy, experimental
- `vault/personal/` — your curated notes, never touched by the tool

---

## Install

```bash
git clone https://github.com/Ray0907/wiki-compiler
cd wiki-compiler
uv sync
export ANTHROPIC_API_KEY=sk-ant-...
```

## Usage

```bash
# Compile a source into your vault
uv run wiki.py add https://github.com/kepano/obsidian-skills

# Written: vault/agent/kepanoobsidian-skills-agent-skills-for-obsidian.md

# Ask a question against everything you've compiled
uv run wiki.py query "What is the Obsidian CLI used for?"
```

Then open `vault/` as an Obsidian vault — every `[[wikilink]]` in the compiled pages becomes a node in your knowledge graph.

---

## Output format

Each compiled page:

```markdown
---
title: kepano/obsidian-skills
url: https://github.com/kepano/obsidian-skills
summary: A collection of agent skill files that teach AI agents how to read
  and write Obsidian vaults using Markdown, Bases, JSON Canvas, and CLI.
categories:
  - AI
  - Tools
tags:
  - obsidian
  - agents
  - mcp
date_compiled: '2026-04-03'
---
## Summary
A collection of agent skill files...

## Key Concepts
- [[Obsidian CLI]]
- [[Agent Skills]]
- [[JSON Canvas]]
- [[Model Context Protocol]]
- [[Vault Structure]]
```

---

## Schema

`schema.yaml` controls what the compiler produces. Edit it to match your domain:

```yaml
categories:
  - AI
  - Tools
  - Research
  - Knowledge Management
  - Agent Architecture

backlinks:
  min_concepts: 2
  max_concepts: 8
  format: "[[{concept}]]"

output:
  filename_pattern: "{slug}.md"
  frontmatter_fields:
    - title
    - url
    - summary
    - categories
    - tags
    - date_compiled
```

Schema is re-read on every run — no restart needed.

---

## MCP Server

Connect to Claude Desktop, Cursor, or any MCP client:

```bash
uv run mcp_server.py
```

Two tools exposed:
- `add_document(url)` — compile a URL into the vault
- `search_wiki(query)` — query the vault

**Claude Desktop config** (`claude_desktop_config.json`):

```json
{
  "mcpServers": {
    "wiki": {
      "command": "/path/to/wiki-compiler/.venv/bin/python",
      "args": ["/path/to/wiki-compiler/mcp_server.py"],
      "cwd": "/path/to/wiki-compiler",
      "env": {
        "ANTHROPIC_API_KEY": "sk-ant-..."
      }
    }
  }
}
```

---

## Tests

```bash
uv run pytest tests/ -v
# 16 passed in 0.45s
```

---

## Why not an Obsidian plugin?

Obsidian plugins can't run heavy LLM pipelines, are tied to one app, and the Obsidian team is already building native AI. The value here is the **compiler** — the pipeline that turns raw sources into a coherent, schema-consistent knowledge structure. Obsidian is just a convenient viewer for the output.

The MCP server makes the compiled vault available to any LLM client. As models improve, the compilation gets better — the schema and accumulated knowledge stay yours.

---

## Roadmap

- [ ] `wiki compile` — batch mode, multiple sources at once
- [ ] Human review gate — promote pages from `vault/agent/` to `vault/personal/`
- [ ] `wiki report` — autonomous mode: one query spawns a team of agents, builds an ephemeral wiki, returns a full report
- [ ] Obsidian CLI integration — use backlink graph for smarter query routing