# KnowledgeKeeper

> **Every fact your AI knows has a git commit hash on it.**

KnowledgeKeeper is a local-first, open-source GitOps pipeline that turns your team's Slack messages and Notion pages into a versioned knowledge wiki — automatically served to AI coding tools via MCP.

---

## The Problem

AI coding assistants (Claude Code, Cursor, Copilot) are only as good as the context they receive. Your team makes decisions in Slack threads and Notion pages all day — but none of that ever makes it into your AI's context window. The result: Claude suggests a deprecated API endpoint. Cursor still thinks you're using the old database. Nobody updated the rules file after the migration.

**This is not an AI problem. It's a context maintenance problem.**

---

## How It Works

```
Slack / Notion / Jira
        │
        ▼
┌───────────────────────────────────────────────────────────────────┐
│  STAGE A  Full Pull + Triage                                      │
│  Pulls everything since last sync — no keyword filtering.        │
│  Hard filter removes bots, duplicates, short messages.           │
└──────────────────────────┬────────────────────────────────────────┘
                           │
                           ▼
┌───────────────────────────────────────────────────────────────────┐
│  STAGE B  Batch LLM Signal Detection                              │
│  1 LLM call per 25 messages. Detects: decisions, policies,       │
│  architecture choices, metrics, risks.                           │
└──────────────────────────┬────────────────────────────────────────┘
                           │
                           ▼
┌───────────────────────────────────────────────────────────────────┐
│  STAGE C  Wiki Reconciliation                                     │
│  Maps each signal against your existing wiki. Classifies as:     │
│  NEW · UPDATE · SUPPLEMENT · CONTRADICT                          │
└──────────────────────────┬────────────────────────────────────────┘
                           │
                           ▼
┌───────────────────────────────────────────────────────────────────┐
│  STAGE D  Compile to Staging Branch                               │
│  Writes structured OKF Markdown files to a git staging branch.  │
│  Assigns review lanes: AUTO · QUICK · ATTN                       │
└──────────────────────────┬────────────────────────────────────────┘
                           │
                           ▼
┌───────────────────────────────────────────────────────────────────┐
│  STAGE E  Human Review Digest  (Streamlit UI)                    │
│  Reviewer approves proposals in under 10 minutes.                │
│  Approved content merges to main with a git commit hash.         │
└──────────────────────────┬────────────────────────────────────────┘
                           │
                           ▼
         MCP Server — serves wiki to Claude Code, Cursor, Windsurf
```

---

## Key Features

| Feature | Detail |
|---|---|
| **Full-coverage pull** | Pulls everything since last sync — no missed decisions from bad search queries |
| **Wiki reconciliation** | AI maps new signals against existing knowledge before asking a human |
| **Three-lane review** | AUTO (high confidence) · QUICK (one-click) · ATTN (contradictions) |
| **Git audit trail** | Every approved fact has a commit hash, reviewer name, and timestamp |
| **OKF compliance** | Implements Google Cloud's Open Knowledge Format (Apache 2.0, June 2026) |
| **MCP serving** | One server for Claude Code, Cursor, Windsurf, and all MCP-compatible IDEs |
| **Local-first** | Runs entirely on your machine. Zero data egress with Ollama. |
| **Connector adapters** | Pluggable source adapters — all produce a common `RawItem` stream |

---

## Tech Stack

| Layer | Technology |
|---|---|
| LLM Integration | [Pydantic AI](https://ai.pydantic.dev/) — structured JSON extraction with auto-retry |
| Git Operations | [GitPython](https://gitpython.readthedocs.io/) — staging branch, merge, revert |
| MCP Server | [FastMCP](https://github.com/jlowin/fastmcp) — 3 tools served to AI IDEs |
| Review UI | [Streamlit](https://streamlit.io/) — three-lane digest with undo |
| Scheduler | [APScheduler](https://apscheduler.readthedocs.io/) — nightly pipeline + morning notify |
| Source Connectors | slack-sdk · notion-client |
| State Store | SQLite (stdlib) — sync timestamps and run health |
| CLI | Click + Rich |
| Default LLM | qwen3:4b via Ollama (local) · Claude Haiku 4.5 (cloud option) |

---

## Project Structure

```
knowledgekeeper/
├── cli.py                  # Guided setup wizard + start/status commands
├── config.py               # Config dataclass, load/save YAML
├── connectors/
│   ├── base.py             # RawItem dataclass + ConnectorBase ABC
│   ├── slack_connector.py  # Full channel history pull
│   ├── notion_connector.py # Database page pull
│   └── triage.py           # Dedup + length + bot filter
├── pipeline/
│   ├── detector.py         # Stage B: batch LLM signal detection
│   ├── reconciler.py       # Stage C: wiki reconciliation (Pydantic AI)
│   ├── aggregator.py       # Merge proposals for same concept
│   ├── compiler.py         # Stage D: write to staging branch
│   └── runner.py           # Orchestrate A → B → C → D
├── okf/
│   ├── schema.py           # OKFConcept dataclass + frontmatter parse/render
│   ├── index_builder.py    # Rebuild index.md files
│   └── log_writer.py       # Append to audit log
├── git_ops/
│   └── manager.py          # Staging branch, merge, revert, read
├── digest/
│   └── ui.py               # Streamlit review UI (Proposals · Status · Settings)
├── mcp/
│   └── server.py           # FastMCP: get_knowledge_map, read_concept, get_changes
├── db/
│   └── store.py            # SQLite: sync state + run history
└── scheduler.py            # Nightly 22:00 pipeline + 08:00 notify
```

---

## Quick Start

```bash
# 1. Install
git clone https://github.com/YOUR_USERNAME/knowledgekeeper
cd knowledgekeeper
python3 -m venv venv && source venv/bin/activate
pip install -e .

# 2. Pull the local LLM (or skip and use Anthropic)
ollama pull qwen3:4b

# 3. Guided setup — connects Slack + Notion, initialises wiki repo
knowledgekeeper init

# 4. Open the review digest
python3 -m streamlit run knowledgekeeper/digest/ui.py --server.port 8080

# 5. Start the nightly scheduler
knowledgekeeper start
```

**Run the full pipeline once without waiting for the scheduler:**
```bash
python3 demo/run_pipeline.py
```

**Run the happy-path test with synthetic data (no real credentials needed):**
```bash
python3 scripts/run_test.py
```

---

## MCP Integration

Add this to your Claude Code or Cursor MCP config:

```json
{
  "mcpServers": {
    "knowledgekeeper": {
      "command": "python3",
      "args": ["-m", "knowledgekeeper.mcp.server"]
    }
  }
}
```

Your AI now has three tools:
- `get_knowledge_map` — returns the root index of all approved concepts
- `read_concept(path)` — reads a specific OKF file on demand
- `get_changes(since?)` — returns the audit log

---

## The OKF Wiki Output

Every approved proposal becomes a structured Markdown file:

```markdown
---
type: concept.decision
title: Supabase as Primary Database
domain: architecture
confidence: high
proposal_type: NEW
source_refs:
  - author: alex.chen
    channel: engineering
    platform: slack
approved_by: Sarah Okafor
git_commit: a1b2c3d
---

## Decision

The team selected Supabase as the primary database after evaluating
Supabase, Firebase, and PlanetScale...
```

With a root index and audit log:

```
~/my-wiki/
├── index.md          ← knowledge map (what MCP reads first)
├── log.md            ← full audit trail with commit hashes
└── okf/
    ├── architecture/ ← index.md + concept files
    ├── product/
    └── operations/
```

---

## Test Coverage

58 tests across all pipeline stages and components:

```bash
pytest tests/ -v
# 58 passed in 3.2s
```

```
tests/
├── test_config.py           test_git_manager.py
├── test_store.py            test_index_builder.py
├── test_slack_connector.py  test_index_scanner.py
├── test_notion_connector.py test_okf_schema.py
├── test_triage.py           test_okf_writer.py
├── test_detector.py         test_compiler.py
├── test_reconciler.py       test_runner.py
├── test_aggregator.py       test_scheduler.py
├── test_digest_ui.py        test_mcp_server.py
```

---

## Key Design Decisions

**Full pull over keyword search** — Stage A pulls everything since last sync. A keyword search misses decisions phrased in ways we didn't anticipate. The LLM in Stage B is cheap enough that filtering happens there, not at the query layer.

**Wiki reconciliation before human review** — The AI does the cognitive work of mapping new information against existing knowledge. The human makes a binary judgment on a structured proposal — not a raw Slack message.

**Git as the knowledge store** — Proposals land on a `kk-staging` branch. Approval merges to `main`. Every fact is traceable: `git log --oneline okf/architecture/auth-api.md` shows every time that concept was modified and who approved it.

**Connector adapter pattern** — All sources implement `ConnectorBase` and produce `list[RawItem]`. The pipeline from Stage B onwards is source-agnostic. Adding Jira or Linear requires one new file in `connectors/`.

**No LangChain, no LlamaIndex** — Pydantic AI for structured LLM calls. GitPython for git. FastMCP for serving. Everything else is stdlib Python.

---

## Roadmap

- [x] Slack + Notion connectors
- [x] 5-stage pipeline (triage → detect → reconcile → compile → review)
- [x] Streamlit digest UI with three review lanes
- [x] MCP server (Claude Code, Cursor, Windsurf)
- [x] Guided CLI setup wizard with live token validation
- [x] Settings tab in Streamlit UI
- [x] OKF-compliant wiki structure (index.md + log.md)
- [ ] Jira connector
- [ ] PyPI package release
- [ ] OAuth web flow for Slack + Notion setup
- [ ] Cost dashboard

---

## License

Apache 2.0 — use it, fork it, build on it.

---

*Built as an AI PM portfolio project. If you're solving context drift on your team, I'd love to hear from you.*