wiki-context
by Minokun
README.md
# wiki-context
**The Deterministic Context Engine for AI Agents** — Offline Compiled LLM Wiki + Table-Aware SQLite FTS5 + Zero-Hallucination MCP Gateway.
English | [简体中文](README.zh-CN.md)
[](LICENSE)
[](pyproject.toml)
[](pyproject.toml)
[](docs/gateway-and-acl.md)
---
## Why `wiki-context`? (Core Value)
Modern developers and teams already work inside capable AI agents (**Claude Code, Codex CLI, Cursor, DeepSeek Harness**). They do **not** need another bloated RAG chat webapp backed by an 8 GB Docker stack (vector DB + embedding service + reranker + Postgres + rigid Web UI).
What agents actually lack is a **millisecond-latency, deterministic, citation-anchored librarian**:
- **Decoupled Write vs. Read Paths**: Heavy comprehension happens **offline** (native document extraction + AI-compiled cross-document Wiki synthesis + SHA-256 freshness tracking). Online retrieval uses **zero runtime LLM calls**, **zero external APIs**, and **zero third-party Python packages**.
- **One Call, Complete Context (`~2–35 ms`)**: A single `kb_context` tool call deterministically assembles **high-level Wiki synthesis (24% budget)** + **1-hop provenance-verified primary source excerpts with pinned table headers (76% budget)** + **explicit missing-concept hallucination guardrails**.
- **Clear Architectural Boundary (`Context` vs. `State & Action`)**: `wiki-context` serves read-only ground truth — RFCs, ADRs, product specs, pricing tiers, SOPs, and postmortems. Pair it alongside transactional MCP servers (SQL / ERP / GitHub / Jira MCP) so your agent reads the rules from `wiki-context` in 3 ms before querying live state or executing mutations.
```text
Raw Sources (`sources/` or `史料/` — PDF, DOCX, PPTX, XLSX, CSV, MD; Strictly Read-Only)
│ 1. wctx sync (Zero-dep native extraction + pluggable CLI OCR + SHA-256 manifest)
▼
Derived Mirrors (`mirrors/` or `史料_md/`) Compiled Wiki (`wiki/` or `知识库/`)
│ (Page/line/sheet locators) │ (Agent/Obsidian curated or `wctx compile`
│ │ + SHA-256 `source_hashes` + `wctx lint`)
└────────────────────┬───────────────────────┘
│ 2. wctx index (Table-aware chunks + dup folding + freshness check)
▼
Immutable `gen-<id>/bm25.sqlite` ◄── Atomic `current.json` swap (lock-free `?immutable=1`)
│
│ 3. One-shot `kb_context` (~2–35 ms, 0 runtime LLM tokens)
▼
Claude Code / Codex / Cursor / DSH / Custom Agents (Stdio MCP, Streamable HTTP `/mcp`, REST)
```
---
## `wiki-context` vs. Traditional RAG vs. Pure LLM Wiki
Today's agent knowledge stacks generally fall into two camps — **Traditional Chunk/Vector RAG** and **Pure Agent-Browsed LLM Wiki** (e.g., the Karpathy-style Markdown Wiki pattern). Both break down on real engineering and enterprise workloads, for opposite reasons:
### 1. Why Traditional Chunk + Vector RAG Falls Short
1. **Embedding Blur on Exact Identifiers**: Dense embeddings smooth over exact technical codes (`RFC-042`, `TB/T-2075`, `HTTP-429`, SKUs, part numbers, version flags), retrieving semantically "similar" chunks for the wrong component or tier.
2. **Headerless Table Slicing**: Fixed-size chunkers (e.g., 500–800 chars) slice multi-hundred-row CSV, Excel, and Markdown tables mid-sheet. Row 80 matches the query, while Row 1's column headers are left behind — leaving the LLM to guess what `[12000, 36000, 64]` mean.
3. **Top-K Myopia & Mega-Manual IDF Flattening**: Isolated chunks lack cross-document architectural evolution ("which ADR superseded RFC-012 and why?"). Meanwhile, a single 300-page manual can flood chunk-level term frequencies and crowd out concise specs.
4. **Nearest-Neighbor Hallucination & Infra Bloat**: Even when a queried concept (`Kafka`) does not exist anywhere in the vault, vector search still returns the top-K closest neighbors, nudging the agent to fabricate answers — all while requiring external vector DBs and embedding APIs.
### 2. Why Pure LLM Wiki (Agent-Browsed Markdown) Hits Production Bottlenecks
The "LLM Wiki" pattern replaces vector DBs with LLM-maintained Markdown synthesis pages (`wiki/*.md` + `index.md`) that agents browse via `read`/`grep`. It nails global narrative, but introduces four production pain points:
1. **Multi-Turn Latency & 10× Token Burn**: Answering one question often takes **3–6 tool rounds** (`read index.md` → open 2–3 Wiki pages → `grep` raw files for missing figures), burning **15–40 seconds** and **30k–40k context tokens** before the agent even starts solving your task.
2. **Lossy Compression Trap**: Wiki summaries excel at structure and trade-offs, but inevitably drop long-tail clauses and individual rows of a 500-row quota or BOM spreadsheet. Reading only the Wiki loses exact numbers; grepping only raw files loses the big picture.
3. **Silent Staleness (Wiki Drift)**: When an underlying Word spec or CSV table updates a limit from `5,000` to `12,000`, prose Wiki pages silently go stale and actively mislead the agent unless every cited source hash is cryptographically tracked.
4. **Confidentiality Downgrade & No Hard Guardrails**: Exposing a raw Wiki folder to multi-user agents lacks provenance-aware RBAC — a user blocked from `sources/finance/` can inadvertently read confidential figures summarized inside a high-level `wiki/Overview.md` page.
### 3. How `wiki-context` Synthesizes Both Worlds
`wiki-context` treats knowledge like a **compiler + deterministic linker**:
- **Keeps the Soul of LLM Wiki (Plain Markdown + Global Synthesis)**: Human-readable `wiki/*.md` pages with `[[wikilinks]]` (editable in Obsidian, Git-diffable), upgraded with **SHA-256 `source_hashes`** in YAML frontmatter. If any cited source changes by 1 byte, the indexer automatically flags the Wiki page `stale` (`0.5×` score demotion + explicit `⚠️ stale` warning badge in `<kb_context>`).
- **Exceeds RAG Precision in One Deterministic Call**:
- **Code-Preserving Bilingual FTS5 + Per-Source IDF**: Keeps intact identifiers (`RFC-042`, `350mm`, `v2.4`), segments CJK bigrams without external NLP deps, and computes IDF over **distinct source documents (`df_sources`)** rather than chunks.
- **Controlled 1-Hop Wiki Provenance**: High-confidence Wiki hits contribute synthesized guidance **and** boost (`1.25×`) their cited primary documents — *only* when the primary chunk independently matches a high-IDF query concept, preventing broad overview pages from dragging in noise.
- **Header-Discounted Table Windowing**: Oversized table sections are sliced with `table_header` **pinned automatically at the top of every window**, while discounting header tokens during window scoring so deep target rows are never missed.
- **Zero-Hit Anti-Hallucination & Provenance-Inherited ACL**: Unmatched core query terms trigger a mandatory `⚠️ Missing-Concept Alert`. Under `PrefixACLPolicy`, a Wiki page is visible *if and only if* the caller is authorized to read **every** primary document listed in its `sources:`.
### Side-by-Side Comparison
| Dimension | Traditional Vector RAG | Pure LLM Wiki (Agent-Browsed) | **`wiki-context`** |
| :--- | :--- | :--- | :--- |
| **Core Paradigm** | Chunking + Dense Embeddings + Reranker | Markdown Wiki + Multi-turn Agent `read`/`grep` | **Offline Compiled Wiki + Deterministic Dual-Layer FTS5** |
| **Exact Codes & IDs (`RFC-042`, `HTTP-429`)** | Weak (semantic embedding blur) | Exact only if agent guesses right `grep` | **Exact** (code-preserving FTS5 + per-source `df_sources` IDF) |
| **Deep Tables (CSV / XLSX / MD)** | Loses column headers mid-table | Wiki summaries omit long-tail rows | **Pinned `table_header`** on every window + header-discounted slicing |
| **Cross-Document Synthesis** | Weak (isolated top-K fragments) | **Strong** (curated synthesis pages) | **Strong** (24% Wiki synthesis + 76% 1-hop primary ground truth) |
| **Colloquial -> Domain Jargon** | Fuzzy vector proximity (black-box) | Depends on agent guessing keywords | **Hot-reloaded `synonyms.json`** (deterministic 1-way concept merge) |
| **Stale Summary Detection** | N/A (no synthesis layer) | Manual discipline (silent drift risk) | **Automatic SHA-256 `source_hashes`** (`stale` badge + `0.5×` demotion) |
| **Unanswerable Query Defense** | High hallucination risk (feeds nearest chunks) | Depends on agent stopping on time | **Explicit `⚠️ Missing-Concept Alert`** + unmirrored boundary notes |
| **Online Latency & Token Cost** | 1–5 s + Embedding/Rerank API calls | 15–40 s (3–6 tool calls, ~30k–40k tokens) | **1 tool call, ~2–35 ms, 2.6k–4.8k tokens, 0 LLM calls** |
| **Access Control (RBAC)** | Chunk metadata filter (often leaky) | File-level OS permissions only | **Hot-reloaded `PrefixACLPolicy` + 100% Wiki provenance inheritance** |
| **Runtime Footprint** | Heavy (Milvus/Qdrant + Docker + SDKs) | Plain folder | **Zero dependencies** (Python 3.10+ stdlib + immutable SQLite FTS5) |
---
## What One `kb_context` Call Actually Returns
### Case 1: Deep CSV Table Lookup + Wiki Provenance (`~3 ms`)
Query: *"What are the Sustained_RPS, Burst_Bucket, and Max_Payload_MB limits for the Enterprise-Dedicated tier?"*
```markdown
<kb_context vault="AuroraEdge Engineering KB" route="fact" generation="gen-20261122T133553Z-1486c582">
## 1. Compiled Wiki Synthesis (Verify exact figures & clauses against Primary Sources below)
### [W1] wiki/Gateway-Rate-Limiting-and-SLOs.md § Key Facts (L20-L24) [freshness:fresh]
- **Token Bucket Enforcement**: Edge nodes enforce per-tenant token buckets using local atomic counters synced every `250ms`...
- **Enterprise Dedicated Tier**: Exact per-tier RPS and burst ceilings are defined in `sources/specs/tier-quotas.csv`.
## 2. Primary Source Evidence (Authoritative Ground Truth)
### [S1] sources/specs/tier-quotas.csv | tier-quotas | L24-L47 (section L8-L48) [via Wiki provenance]
| Tier_Code | Sustained_RPS | Burst_Bucket | Max_Payload_MB | Support_SLA |
| --- | --- | --- | --- | --- |
...
| Scale-05 | 5000 | 12500 | 25 | 24x7-Priority |
| Enterprise-Dedicated | 12000 | 36000 | 64 | 24x7-15min-Page |
## 3. Gap Diagnostics & Grounding Guardrails (Mandatory)
- Grounding Rule: Every claim must be directly supported by [S1]~[Sn] above with explicit source citations...
</kb_context>
```
Notice how `[S1]` slices lines `L24–L47` deep inside `tier-quotas.csv` while **pinning the CSV header row automatically** at the top, boosted via `[W1]`'s verified provenance.
### Case 2: Unanswerable Question Guardrail (`~0.2 ms`)
Query: *"What is the Kafka partition retention period for billing events?"* (when the vault uses Raft/WAL, not Kafka)
```markdown
<kb_context vault="AuroraEdge Engineering KB" route="fact" generation="gen-20261122T133553Z-1486c582">
## 2. Primary Source Evidence (Authoritative Ground Truth)
*(No directly matching primary source chunks found)*
## 3. Gap Diagnostics & Grounding Guardrails (Mandatory)
- ⚠️ Missing-Concept Alert: Key query concept(s) 'Kafka', 'partition', 'retention', 'period', 'billing', 'events' had 0 hits across all retrieved chunks! This likely indicates a knowledge gap in the vault — do NOT hallucinate figures or clauses from pretraining.
</kb_context>
```
---
## Key Engineering Highlights
- **Zero Runtime & Build-Time Dependencies**: Pure Python 3.10+ standard library — including a self-contained PEP 517/660 wheel backend, native XML/ZIP extractors for `.docx`, `.pptx`, `.xlsx`, `.csv`/`.tsv`, `.md`/`.txt`, `pdftotext` integration for `.pdf`, and a pluggable `WCTX_OCR_CMD` hook for scanned pages.
- **Native English & Chinese Vault Layouts**:
- English (`--layout en`): `sources/`, `mirrors/`, `wiki/`, `index.md`, `log.md`
- Chinese Obsidian (`--layout zh`): `史料/`, `史料_md/`, `知识库/`, `目录.md`, `日志.md` (auto-detected with zero config on existing Obsidian LLM Wiki vaults).
- **Lock-Free Concurrency & Atomic Hot Reload**: Readers open versioned `gen-<id>/bm25.sqlite` snapshots via `?mode=ro&immutable=1` on thread-local connections. Publishing a new index atomically swaps `current.json` and drains old snapshots over a 30-second grace window with zero dropped requests.
- **Defense-in-Depth Security**: Automatic credential scrubbing (`PRIVATE KEY`, `sk-...`, `AKIA...`, `Bearer`, `Cookie`) before snippets leave the server, `O_NOFOLLOW` symlink-safe file reads, sensitive-vault guardrails (`sensitive: true`), and multi-vault Streamable HTTP RBAC.
---
## 60-Second Quickstart
### 1. Install
```bash
git clone https://github.com/Minokun/wiki-context.git
cd wiki-context
pip install -e .
```
*(Zero third-party packages are downloaded or installed. Optional: install `poppler` if you need native `pdftotext` PDF extraction.)*
### 2. Run the Reproducible Benchmark
```bash
python3 examples/benchmark/run_benchmark.py
```
Output on `examples/demo-vault`:
```text
ID Route NaiveBM25@3 ContextPack Latency(ms) Question
----------------------------------------------------------------------------------------
Q1 fact MISS PASS 2.94 What are the Sustained_RPS, Burst_Bucket, and
Q2 fact PASS PASS 3.54 When requests get throttled at the edge, how o
Q3 fact PASS PASS 1.87 How fast is container warmup p95 latency and h
Q4 complex PASS PASS 2.13 Compare Gateway circuit-breaker trip threshold
Q5 fact (halluc.risk) GUARD_OK 0.20 What is the Kafka partition retention period f [Unanswerable]
----------------------------------------------------------------------------------------
Strict Complete-Evidence Recall : Naive 600c BM25 = 3/4 (75.0%) vs ContextPack = 4/4 (100.0%)
Unanswerable Gap Guardrail : 1/1 (100.0%)
ContextPack Median Latency : 2.13 ms
```
### 3. Initialize Your Own Vault
```bash
# English layout (`sources/`, `mirrors/`, `wiki/`, `index.md`)
wctx init ./my-kb --name "Platform Engineering KB"
# Or Chinese Obsidian layout (`史料/`, `史料_md/`, `知识库/`, `目录.md`)
wctx init ./公司知识库 --name "研发知识库" --layout zh
# Drop PDFs, DOCX, XLSX, PPTX, CSV, or Markdown into `sources/`, then sync:
cp ~/Specs/*.docx ./my-kb/sources/
wctx sync --vault ./my-kb
# Test one-shot context assembly in terminal:
wctx context --vault ./my-kb -q "What are our API retry and rate-limit rules?"
```
---
## Vault Structure & How to Organize Your Knowledge Base
When adopting a knowledge engine, the most practical questions are: **"What does the folder structure actually look like?"** and **"Which files do I manage by hand vs. what gets generated automatically?"**
`wiki-context` enforces a clean **three-layer directory contract**, supporting both English (`--layout en`) and Chinese Obsidian (`--layout zh`) naming out of the box.
### 1. Annotated Vault Directory Tree
Below is a complete vault initialized with `wctx init ./my-kb --layout en` (Chinese `--layout zh` equivalents shown in parentheses):
```text
my-kb/ # Vault Root
├── vault.json # [Config] Name, layout, max_doc_bytes, excludes, OCR dirs, local ACL
├── synonyms.json # [Config] Colloquial aliases -> canonical domain terms (hot-reloaded on save)
├── AGENTS.md # [Generated] Operating rules telling Claude Code / Codex / DSH how to curate
├── index.md (目录.md) # [Catalog] Topic navigation map registering [[Wiki Pages]] (Obsidian home)
├── log.md (日志.md) # [Audit Log] Append-only timeline of init / sync / compile operations
│
├── sources/ (史料/) # [Layer 1: Raw Ground Truth — YOU drop files here (Strictly Read-Only)]
│ ├── 01-public-specs/ # ├─ Organize freely into nested folders by domain, team, or security tier
│ │ ├── rfc-042-edge-gateway.md # ├─ Text & tables: .md, .txt, .rst, .csv, .tsv
│ │ └── platform-arch-v2.docx # └─ Office & PDF: .docx, .pptx, .xlsx, .pdf (via poppler pdftotext)
│ ├── 02-quotas-and-pricing/
│ │ └── tier-quotas.xlsx
│ ├── 03-diagrams-and-scans/ # (Optional: whitelist in `ocr_image_dirs` + `WCTX_OCR_CMD` for image OCR)
│ ├── 90-confidential-finance/ # (Maps 1:1 to `PrefixACLPolicy` directory-prefix RBAC rules)
│ └── 99-deprecated-archive/ # (Add to `exclude_patterns` in vault.json to mute legacy docs without deleting)
│
├── mirrors/ (史料_md/) # [Layer 2: Derived Mirrors & Index — Auto-generated by `wctx`, DO NOT edit]
│ ├── .manifest.json # ├─ Incremental SHA-256 & mtime extraction manifest
│ ├── 01-public-specs/ # ├─ Mirrors `sources/` 1:1 with extracted Markdown + locator sidecars:
│ │ ├── platform-arch-v2.docx.md # │ ├─ Extracted text & full Markdown tables
│ │ └── platform-arch-v2.docx.json # │ └─ Page / slide / sheet line-range metadata
│ └── .index/ # └─ Versioned immutable search snapshots (safe to `.gitignore`)
│ ├── current.json # ├─ Atomic pointer to active generation (zero-downtime hot-swap)
│ └── gen-20261122T133553Z-xxxx/ # └─ Immutable `bm25.sqlite` + JSONL corpus artifacts
│
└── wiki/ (知识库/) # [Layer 3: Compiled Wiki — Flat `*.md` cross-document synthesis pages]
├── Overview.md (知识库总览.md) # ├─ Root entrypoint page
├── Gateway-Rate-Limiting.md # ├─ Topic synthesis (declares YAML `sources:` & `source_hashes:`)
└── Compute-Runtime-and-DR.md # └─ Linked via flat `[[Gateway-Rate-Limiting]]` wikilinks
```
| Layer / File | Maintained By | Role & Golden Rule | Commit to Git? |
| :--- | :---: | :--- | :---: |
| **`sources/` (`史料/`)** | **Humans only** | Immutable ground truth; **never modified, renamed, or deleted** by `wctx` or agents | Yes for Markdown/CSVs; Git LFS or shared storage for large binaries |
| **`mirrors/` (`史料_md/`)** | **`wctx sync` (Auto)** | Derived Markdown mirrors, page locators, and `bm25.sqlite`; 100% reproducible | **Ignore `.index/`** (or ignore `mirrors/` entirely) |
| **`wiki/` (`知识库/`)** | **Agent / Human / `wctx compile`** | Cross-document architecture synthesis, trade-offs & caveats; **keep flat (`wiki/*.md`)** | **Yes** (plain Markdown, easy PR review) |
| **`index.md` / `vault.json` / `synonyms.json`** | **Agent / Human** | Topic hierarchy catalog, vault policy config, and colloquial synonym map | **Yes** |
---
### 2. Step-by-Step Guide: Creating & Maintaining a Vault
#### Step 1: Scaffold a New Vault (or Mount an Existing Folder)
- **Brand-new vault**: Pass a **non-existent or empty** directory path to `wctx init` (non-empty folders are safely refused so existing files are never overwritten):
```bash
wctx init ./my-kb --name "Platform Engineering KB" --layout en
# Or Chinese Obsidian layout:
wctx init ./研发知识库 --name "研发技术规范库" --layout zh
```
- **Existing folder (Zero-Config Auto-Detection)**: Already have a folder containing `sources/` (or `史料/` / `知识库/`)? You don't even need `wctx init` or `vault.json` — run `wctx sync --vault /path/to/folder` directly and `wiki-context` auto-detects the layout.
#### Step 2: Organize Subfolders Under `sources/` (`史料/`)
Drop your raw files into `sources/`. While flat files work fine, organizing `sources/` into **domain, permission, and lifecycle subfolders** unlocks powerful built-in capabilities:
1. **Folder Hierarchy = RBAC Boundaries**: `PrefixACLPolicy` matches path prefixes directly (e.g., granting contractors `"allow": ["sources/01-public-specs/**"]` and `"deny": ["sources/90-confidential-finance/**"]`). Structuring subfolders cleanly gives you instant zero-refactor access control later.
2. **Archive Without Deleting**: Move superseded specs into `sources/99-deprecated-archive/` and set `"exclude_patterns": ["99-deprecated-archive/"]` in `vault.json` to keep them out of search results while preserving historical records.
3. **Automatic File Admission & Hygiene**:
- **Admitted natively**: `.md`, `.markdown`, `.txt`, `.rst`, `.csv`, `.tsv`, `.docx`, `.pptx`, `.xlsx`, and `.pdf` (when `pdftotext` is installed).
- **Skipped safely**: Credentials & keys (`.env`, `*.pem`, `*.key`, `id_rsa`, `credentials.json`), machine dumps/logs (`.log`, `.json`, `.yaml`, `.sql`, `.sqlite`), media/archives, hidden/asset directories (`.git`, `.obsidian`, `*.assets`), and oversized files (`>2 MiB` by default, configurable via `max_doc_bytes`) are excluded with transparent reasons in `kb_status`. Byte-identical copies (e.g., `spec (1).docx`) are automatically folded during indexing.
- **Targeted Image / Scan OCR**: Inline decorative images are skipped by default; to OCR architecture diagrams or scanned contracts in specific folders, add `"ocr_image_dirs": ["03-diagrams-and-scans"]` in `vault.json` and set `WCTX_OCR_CMD`.
#### Step 3: Run `wctx sync` (Your Vault Is Immediately Queryable!)
```bash
wctx sync --vault ./my-kb
```
> **Progressive Adoption**: You do **not** need to write any `wiki/*.md` pages on Day 1. As soon as `wctx sync` finishes, all primary documents and spreadsheets are chunked (with table headers preserved) into `bm25.sqlite` and ready for MCP queries.
#### Step 4: Curate Flat `wiki/*.md` Pages & `synonyms.json`
When knowledge about a topic spans multiple raw files (e.g., two RFCs, a CSV quota sheet, and a DR runbook), synthesize a **flat** Markdown page inside `wiki/` (avoid nested subdirectories inside `wiki/`; organize topic hierarchy in root `index.md` via `[[Page-Name]]` links):
- **Anatomy of a Compiled Wiki Page (`wiki/Gateway-Rate-Limiting-and-SLOs.md`)**:
```markdown
---
title: "Gateway-Rate-Limiting-and-SLOs"
type: synthesis
status: reviewed
updated: 2026-11-22
sources:
- sources/01-public-specs/rfc-042-edge-gateway.md
- sources/02-quotas-and-pricing/tier-quotas.xlsx
source_hashes:
"sources/01-public-specs/rfc-042-edge-gateway.md": "aa2a7d5bd5ed..."
"sources/02-quotas-and-pricing/tier-quotas.xlsx": "3fb80b1f5dc2..."
---
# Gateway-Rate-Limiting-and-SLOs
## Key Facts
- Edge workers sync local token-bucket counters every `250ms` and return `HTTP-429` + `Retry-After` on overflow.
- Exact per-tier burst ceilings live in `sources/02-quotas-and-pricing/tier-quotas.xlsx`.
## Related Pages
[[Overview]]; [[Compute-Runtime-and-DR]]
```
*(Note: Never compute `source_hashes` manually — `wctx compile` or `wctx freshness --vault ./my-kb snapshot` writes them automatically.)*
- **Map Colloquial Team Terms in `synonyms.json` (Hot-Reloaded on Save)**:
```json
{
"version": 1,
"groups": [
{
"canonical": ["rate-limit", "HTTP-429"],
"aliases": ["throttled", "too many requests", "429 error"]
}
]
}
```
- **Example `vault.json` Configuration**:
```json
{
"name": "Platform Engineering KB",
"layout": "en",
"max_doc_bytes": 5242880,
"sensitive": false,
"exclude_patterns": ["99-deprecated-archive/"],
"ocr_image_dirs": ["03-diagrams-and-scans"],
"acl": {
"partner": {
"allow": ["sources/01-public-specs/**"],
"deny": ["sources/90-confidential-finance/**"]
}
}
}
```
#### Step 5: Daily Update Lifecycle When Raw Docs Change
Whenever you add new files or overwrite an updated `.docx`/`.xlsx`/`.md` inside `sources/`:
1. Run `wctx sync --vault ./my-kb`: extracts changed files incrementally. If any existing Wiki page cited a modified source file, the indexer automatically detects the SHA-256 mismatch and marks that Wiki page `stale` (`0.5×` ranking demotion + `⚠️ stale` warning badge) so outdated prose never silently misleads your agent.
2. Refresh affected Wiki pages (pick either):
- **Automated CLI**: Run `wctx compile --vault ./my-kb` to compile new mirrors, refresh `stale` Wiki pages, snapshot hashes, run `lint`, and hot-swap `bm25.sqlite`.
- **Agent-in-the-Loop**: Ask Claude Code / Codex / DSH to update the relevant `wiki/*.md` page and `index.md`, then seal it with:
```bash
wctx freshness --vault ./my-kb snapshot
wctx lint --vault ./my-kb
wctx index --vault ./my-kb
```
Running `wctx serve` or `wctx gateway` instances automatically hot-swap to the new snapshot on the very next request with zero downtime.
---
## Connect to Your AI Agent (MCP)
`wiki-context` exposes **6 focused, read-only MCP tools** (with dynamic `[vault_name]` disambiguation when multiple vaults are mounted simultaneously):
| MCP Tool | Purpose |
| :--- | :--- |
| **`kb_context`** | **Preferred default (answers 95% of queries in 1 call)**: assembles Wiki synthesis + primary evidence + table headers + gap alerts |
| `kb_search` | Fine-grained BM25/FTS5 search with optional `section` / `page` / `wiki` unit filtering |
| `kb_read` | Sandboxed exact line-range (`<=400` lines) or `page_num` reader over mirrors, Wiki pages, and `index.md` |
| `kb_route` | Classifies query complexity (`fact` 2.6k budget vs. `complex` 4.8k budget) |
| `kb_list` | Lists indexed documents and extraction methods with path filtering |
| `kb_status` | Reports active index generation, document/chunk counts, skipped files, and coverage boundaries |
### Option A: Local Stdio MCP (Single User)
**Claude Code (`~/.claude.json`) / Cursor (`~/.cursor/mcp.json`) / DeepSeek Harness**:
```json
{
"mcpServers": {
"kb": {
"command": "wctx",
"args": ["serve", "--vault", "/absolute/path/to/my-kb"],
"env": { "KB_USER": "alice" }
}
}
}
```
**Codex CLI (`~/.codex/config.toml`)**:
```toml
[mcp_servers.kb]
command = "wctx"
args = ["serve", "--vault", "/absolute/path/to/my-kb"]
env = { KB_USER = "alice" }
```
### Option B: Multi-User, Multi-Vault Remote HTTP Gateway (Teams)
Host multiple vaults behind a single lightweight process with SHA-256 Bearer tokens and hot-reloaded directory-prefix RBAC (`PrefixACLPolicy`):
```bash
WCTX_TOKENS="alice:secret_tok_1,bob:secret_tok_2" \
wctx gateway \
--vault eng=./eng-kb \
--vault policy=./policy-kb \
--host 127.0.0.1 --port 8090
```
- **Streamable HTTP MCP (`2025-03-26`)**: `POST http://127.0.0.1:8090/mcp` (default) or `POST http://127.0.0.1:8090/mcp/eng`, `/mcp/policy`
- **REST JSON API**: `POST /v1/context`, `POST /v1/read`, `GET /v1/vaults`, `GET /v1/whoami`, `GET /healthz`
- **Stdio-to-HTTP Bridge**: Clients that only speak local stdio can connect via `wctx bridge --url https://kb.example.com/mcp/eng --token <TOKEN>`.
---
## Offline Wiki Compilation Workflows
You can build and maintain `wiki/*.md` pages in either of two ways (or mix both):
1. **Interactive Agent + Obsidian Curation**: Open Claude Code / Codex / DSH inside your vault and ask it to organize new documents. Guided by the scaffolded `AGENTS.md`, your agent runs `wctx sync` -> updates `wiki/*.md` & `index.md` -> snapshots `source_hashes` -> runs `wctx lint` -> rebuilds FTS5.
2. **Batch CLI Compiler (`wctx compile`)**: Run the built-in zero-SDK compiler against any OpenAI-compatible (`/v1/chat/completions`, including DeepSeek / local vLLM) or Anthropic (`/v1/messages`) endpoint:
```bash
# Preview uncompiled mirrors and stale Wiki pages without calling an LLM
wctx compile --vault ./my-kb --dry-run
# Compile new pages, refresh stale pages, snapshot SHA-256 hashes, lint, and hot-swap FTS5
WCTX_API_KEY="sk-..." \
WCTX_BASE_URL="https://api.openai.com/v1" \
WCTX_MODEL="gpt-4o-mini" \
wctx compile --vault ./my-kb
```
---
## CLI Reference (`wctx`)
| Command | Description |
| :--- | :--- |
| `wctx init <path> [--layout en\|zh]` | Initialize a three-layer vault (`sources/`, `mirrors/`, `wiki/`) |
| `wctx sync --vault <V> [--list]` | Incrementally extract changed documents & rebuild FTS5 |
| `wctx index --vault <V> [--check --deep]` | Build or verify immutable `gen-<id>/bm25.sqlite` snapshot |
| `wctx context --vault <V> -q "<Q>"` | Assemble a single-call citation-anchored `<kb_context>` pack |
| `wctx search --vault <V> -q "<Q>"` | Run raw BM25/FTS5 search or `--route` complexity classification |
| `wctx compile --vault <V> [--dry-run]` | Compile unindexed mirrors & refresh `stale` Wiki pages via LLM API |
| `wctx freshness --vault <V> [check\|snapshot]` | Detect `stale` Wiki pages when underlying primary sources change |
| `wctx lint --vault <V>` | Verify broken `[[wikilinks]]`, missing sources, and catalog drift |
| `wctx serve --vault <V> [--test]` | Run local stdio MCP server (6 tools, secret-scrubbed) |
| `wctx gateway --vault id=path [--test]` | Run multi-user, multi-vault Streamable HTTP MCP + REST gateway |
| `wctx bridge --url <URL> --token <T>` | Bridge local stdio MCP clients to a remote HTTP gateway |
---
## Documentation
- [Architecture & 5-Step Retrieval Design](docs/architecture.md)
- [Vault Layout, Synonyms & SHA-256 Freshness Spec](docs/vault-spec.md)
- [Multi-Vault Gateway, Authentication & Prefix ACL](docs/gateway-and-acl.md)
## License
[MIT](LICENSE)
This server cannot be deployed
Maintenance
ActivityMaintained
ResponsivenessNo recent community issues