# Ken

A frontend-free memory layer that turns any folder of documents into a searchable knowledge graph. Ken converts, ingests, links, and retrieves your files locally, with no cloud and no runtime dependencies on external repositories.

## Quick Start

Install Ken globally (or run it on demand with npx):

```bash
npm install -g @foresights/ken
# or
npx @foresights/ken <command>
```

Turn any folder into a vault. Files are never moved, copied, or renamed. Ken only creates a `.ken/` directory inside the folder.

```bash
ken init ~/Documents/my-notes
```

Search it:

```bash
ken search "quarterly revenue trends"
```

Results come back in two flavors: **DIRECT** hits (text match) and **GRAPH** hits (linked neurons surfaced by Personalized PageRank).

## Claude Desktop Integration

Ken speaks the Model Context Protocol (MCP) over stdio. Add it to your Claude Desktop config file.

**macOS:** `~/Library/Application Support/Claude/claude_desktop_config.json`
**Windows:** `%APPDATA%\Claude\claude_desktop_config.json`

```json
{
  "mcpServers": {
    "ken": {
      "command": "npx",
      "args": ["-y", "@foresights/ken", "mcp", "--vault", "/path/to/vault"]
    }
  }
}
```

Replace `/path/to/vault` with the absolute path to your Ken vault folder. Restart Claude Desktop and Ken's tool catalog becomes available to the agent.

## CLI Commands

| Command | Description |
|---|---|
| `ken init <path>` | Turn an existing folder of documents into a Ken vault. Creates `.ken/` inside the folder. |
| `ken sync <folder>` | Sync the vault with folder changes. Ingests new and changed files, marks deleted files as orphaned. |
| `ken ingest <file>` | Ingest a single document into the vault. Converts non-text formats automatically. |
| `ken search <query>` | Search the vault. Returns direct text matches plus graph-surfaced results. |
| `ken convert <file>` | Convert a document to Markdown using Ken's native Rust converter. |
| `ken converters` | Check which converters are available (native Rust, OCR, Whisper). |
| `ken serve` | Start the Ken HTTP server (default port 7700, 44 REST routes). |
| `ken mcp` | Start the MCP stdio server for Claude Desktop and other MCP clients. |
| `ken health` | Vault diagnostics. Reports brain health score, graph stats, orphans, stale pages, dead links, contradictions. |
| `ken brief` | Print an orientation packet summarizing the vault. |
| `ken consolidate` | Run the nightly memory consolidation worker (replay, stale-marking, edge decay). |
| `ken review` | List neurons due for spaced repetition review. |
| `ken recall <neuronId>` | Record a memory recall event (updates FSRS state and contribution score). |
| `ken observe` | Run the observation pipeline. Finds neuron clusters for synthesis. |
| `ken expand <query>` | Show the PMI-expanded query for debugging search recall. |
| `ken reindex-pmi` | Rebuild the PMI co-occurrence matrix from all neuron content. |
| `ken models` | Detect local LLM backends (Ollama, LM Studio, llama.cpp) and store selection in `.ken/config.json`. |
| `ken validate` | Validate the vault's `schema.yaml`. |
| `ken tasks` | List conversion tasks by review state (pending, submitted, accepted, rejected, expired). |
| `ken graph export` | Export the knowledge graph as HTML, SVG, GraphML, JSON, or Mermaid. |
| `ken graph affected <neuronId>` | Show reverse impact analysis for a neuron (what depends on it). |
| `ken graph tour <topic>` | Generate a guided knowledge tour starting from a topic. |
| `ken graph benchmark <query>` | Measure token reduction achieved by graph-first retrieval versus raw documents. |
| `ken semantic build` | Build the optional semantic layer (LSI word similarities and node2vec embeddings). Requires Python with numpy. |
| `ken semantic query <word>` | Show semantically similar words from the LSI model. |

## How It Works

Ken has three layers working together.

1. **Rust conversion core.** A native Rust binary converts 11 document formats to Markdown. No system calls to Python, LibreOffice, or external tools. The binary ships inside the npm package.

2. **TypeScript service layer.** The orchestration logic (ingestion, search, graph algorithms, memory lifecycle, MCP server, HTTP server) is written in TypeScript and runs on Node.js.

3. **SQLite storage.** Every vault stores its data in a single `.ken/data.db` SQLite file. Full-text search uses SQLite FTS5. Graph structure, embeddings, FSRS state, and task queues all live in the same database.

The retrieval model is graph-first. When you search, Ken does not just match text. It follows synapses (links between neurons) to surface related knowledge that may not share any keywords with your query.

## Search Architecture

Ken search runs four layers in sequence. Each layer adds recall without sacrificing precision.

1. **FTS5 full-text search.** The base layer. SQLite FTS5 matches your query tokens against neuron content with BM25 ranking. This is always available and requires no model.

2. **PMI query expansion.** Ken builds a Pointwise Mutual Information co-occurrence matrix from all neuron content. When you search, it expands your query terms with their statistical neighbors. Searching "revenue" also hits neurons about "income", "ARR", "MRR". Run `ken reindex-pmi` to rebuild this matrix.

3. **Personalized PageRank.** Seed neurons from layers 1 and 2 become the personalization vector for a PageRank walk over the synapse graph. High-authority neurons connected to your seeds get boosted, even if they had no direct text match. This is what produces the GRAPH results.

4. **Semantic embeddings (optional).** If you run `ken semantic build`, Ken computes LSI word similarities and node2vec graph embeddings using a bundled Python script. This adds latent semantic matching on top of the graph. Requires Python with numpy. Ken works fully without it.

## Memory Lifecycle

Ken treats knowledge like biological memory, not like a static index.

- **FSRS spaced repetition.** Every neuron has an FSRS (Free Spaced Repetition Scheduler) state tracking its stability and retrievability. When you recall a neuron (via `ken recall` or through the MCP tools), its stability increases and its next review date moves out. Neurons you never recall gradually become stale.

- **Hebbian strengthening.** Synapses that get traversed frequently during search and recall get stronger. Infrequently used synapses decay. Run `ken consolidate` to apply this.

- **Nightly consolidation.** The consolidation worker (`ken consolidate`) does three things: (1) replays high-contribution neurons to boost their stability, (2) marks neurons whose retrievability has dropped below the threshold for review, and (3) prunes decayed synapses and edges. This keeps the vault from accumulating dead weight.

- **Observation and synthesis.** The `ken observe` command clusters related neurons and presents them for synthesis. The agent (or you) can then create new neurons that capture higher-level patterns.

- **Self-improvement proposals.** Ken generates typed metric facts with provenance and can detect contradictions. It proposes improvements (new synapses, entity merges, schema updates) as tasks that require human approval before they take effect.

## Supported File Formats

### Native Rust converters (11 formats)

| Format | Extension(s) |
|---|---|
| CSV | `.csv` |
| RTF | `.rtf` |
| HTML | `.html`, `.htm` |
| DOCX | `.docx` |
| XLSX | `.xlsx` |
| PDF | `.pdf` |
| EPUB | `.epub` |
| PPTX | `.pptx` |
| ODT | `.odt` |
| ODS | `.ods` |
| ODP | `.odp` |

### Plain text formats (read directly)

Markdown (`.md`, `.markdown`), plain text (`.txt`), JSON (`.json`), YAML (`.yaml`, `.yml`), XML (`.xml`), HTML (`.html`, `.htm`).

### Code file formats (40+, read as plain text)

TypeScript (`.ts`), JavaScript (`.js`, `.mjs`, `.cjs`), Python (`.py`), Rust (`.rs`), Go (`.go`), Java (`.java`), C (`.c`, `.h`), C++ (`.cpp`, `.cc`, `.cxx`, `.hpp`, `.hh`), C# (`.cs`), Ruby (`.rb`), PHP (`.php`), Swift (`.swift`), Kotlin (`.kt`), Scala (`.scala`), Shell (`.sh`, `.bash`, `.zsh`), SQL (`.sql`), Lua (`.lua`), Perl (`.pl`, `.pm`), R (`.r`, `.R`), Julia (`.jl`), Dart (`.dart`), Clojure (`.clj`), Haskell (`.hs`), Erlang (`.erl`), Elixir (`.ex`, `.exs`), F# (`.fs`), Lisp (`.lisp`, `.cl`), Nim (`.nim`), Zig (`.zig`), Crystal (`.cr`), OCaml (`.ml`), Vue (`.vue`), Svelte (`.svelte`), CSS (`.css`), SCSS (`.scss`), Less (`.less`), TOML (`.toml`), INI (`.ini`, `.cfg`), Dockerfile, Makefile, and more.

## Self-Contained

Ken has zero runtime dependencies on external repositories, cloud services, or API keys.

- **Works offline.** No network calls are required for ingestion, conversion, search, or graph analysis.
- **No cloud.** Your documents never leave your machine. Everything runs locally.
- **No external repos.** The Rust conversion binary ships inside the npm package. The optional semantic layer bundles its own Python script.
- **No API keys.** Ken does not call OpenAI, Anthropic, or any external model. The optional local LLM integration (`ken models`) talks to your own Ollama, LM Studio, or llama.cpp instance on localhost.
- **Single database file.** The entire vault (content, graph, embeddings, FSRS state, task queue) lives in one SQLite file at `.ken/data.db`.

## License

MIT