# amiticia-knowledge-engine

Watches AI/dev content and **writes it into the AmiticIA knowledge repository**
(`amiticia-knowledge`), then DMs a pointer via the existing `whatsapp-mcp`
service. Four entrypoints share the codebase — three one-shot cron jobs that
*write* the repo, and one long-lived server that *reads* it:

- **`watch.py` — mentors.** Curated people we follow (Akita, Lucas Montano,
  Simon Willison). High trust: every item is summarized and scored, no
  relevance gate. Mon/Wed/Fri cron.
- **`research.py` — research feeds.** High-volume, noisy sources (arXiv, Import
  AI, Ahead of AI, AlphaSignal, METR, Epoch). Hard-filtered to 2026+ and
  LLM-sifted for relevance before anything is kept. Own cron.
- **`synthesize.py` — topic synthesis.** Reads the `inbox/` the other two
  wrote, classifies each new entry into one of the fixed `topics/`, and
  rewrites those topic pages as rank-ordered card lists. Runs last in the cron
  chain. The LLM only *classifies*; all ranking and rendering is plain Python,
  so topic pages are reproducible and the LLM never rewrites accumulated
  knowledge.
- **`serve.py` — research MCP server.** A long-lived server (not a cron job)
  that exposes the knowledge repo as two MCP tools — `query_knowledge` (BM25
  retrieval, free) and `research` (the grounded loop: KB + live web + LLM
  synthesis → a dated, cited answer). The tool to consult *before* an
  architecture decision, so the answer comes from current reality.

**Why this exists:** leading-edge AI-agent knowledge moves faster than any
model's training data. This is durable external memory — current, dated,
verified — so architecture decisions are made from 2026 reality, not stale
training.

> **▶ Start here:** [`serve.py`](serve.py) — the read side, and the only
> long-lived process. Everything else exists to fill the corpus it serves.

## Look at first

1. **[`serve.py`](serve.py)** — the research MCP server. Two tools:
   `query_knowledge` (BM25 + a local cross-encoder rerank, free, deterministic,
   no network) and `research` (the grounded loop that returns a JSON contract —
   answer, citations with a **tier** per source, confidence, retrieval date,
   warnings). The tier label is the interesting part: a curated source and an
   open-web result are never presented as equally trustworthy.
2. **[`prompts/`](prompts/)** — every LLM call this system makes, as plain
   markdown files, one per seam: sift, classify, expand, link, research,
   summarize. The design rule is visible here — the LLM only ever *sifts* or
   *classifies*; ranking, rendering and page assembly are plain Python, so topic
   pages are reproducible and no accumulated knowledge is ever LLM-rewritten.
3. **[`promote.py`](promote.py)** — the enrichment loop, and the best example of
   the trust posture. It mines the deep-research history the server persists and
   promotes the *sources* those runs discovered — never the synthesized answer.
   Each candidate URL is re-fetched, sifted for relevance and authority-tagged,
   so a low-trust page is visibly marked rather than silently absorbed.

## What each run does

1. Pull the latest items per source (`sources.py` / `sources_research.py`).
2. Skip items already in the seen-ID state (`state.json` / `research_state.json`).
3. **research.py only:** drop items dated before 2026.
4. LLM-sift each new item with DeepSeek-V3 via OpenRouter:
   - mentors → `prompts/sift_mentor.md` (summary + relevance 1-5, never skipped),
   - research → `prompts/sift_research.md` (summary + relevance, or `SKIP`).
5. Append the keepers as markdown entries to the knowledge repo's
   `inbox/<YYYY-MM-DD>.md` (`knowledge.py`).
6. Persist seen IDs. (The watchers no longer DM — `synthesize.py` is the sole
   digest author; see below. They print feed-health warnings to the cron log.)

The knowledge repo itself lives at `KNOWLEDGE_DIR`; on the VPS that is a git
clone, and the host cron commits + pushes after each run (see Deploy).

### Synthesis (`synthesize.py`)

After the two watchers append to `inbox/`, `synthesize.py` rolls those entries
into the readable `topics/` pages:

1. Parse every `inbox/*.md` entry; skip those already in `synth_state.json`.
2. One LLM call per new entry → a topic slug + optional business *"so what"* +
   optional supersedes flag (`prompts/classify.md`).
3. For each touched topic: parse the existing cards out of the page, add the
   new ones (deduped by link), recompute every card's rank
   (`relevance + recency-decay + authority`, weights in the repo's
   `ranking.toml`), re-sort, rewrite `topics/<slug>.md`.

It only ever writes `topics/`, `AGENTS-BRIEFING.md`, the magazine under `k/`,
`links.json` and `synth_state.json` — never `inbox/`, never git. `--backfill`
reprocesses the whole inbox ignoring state; `--rerank-all` re-ranks every page
without needing new entries (no LLM call).

After ranking, two things happen (both best-effort — a failure never aborts the
canonical topic-page write):

- **Related links.** For each new/changed card, BM25 proposes up to eight
  keyword-similar candidates (`links.py`, reusing the `query.py` retrieval), and
  one bounded LLM call (`prompts/link_related.md`) keeps the 2-3 genuinely
  related ones with a short reason. These are **sidecar** metadata in
  `links.json` (keyed by card link, cached by content hash) — cards are never
  mutated, so the classify-not-rewrite rule holds. The topic pages and magazine
  render a "Related" block from them.
- **Themed digest DM.** `synthesize.py` is now the **sole** WhatsApp digest
  author: it groups this run's new cards by topic (`📚 *Knowledge digest*`, one
  `*Topic*` section each, `• title — so-what` lines, ≤8 with an "…and N more"
  tail), and rides the dead-feed + stale-topic ⚠️ warnings (read from both
  ingest stats files) on the same DM. Sent only when something new was
  classified, **after** which `synth_state.json` and `links.json` are saved —
  the after-send rule, so a failed send replays cleanly.

### Research MCP server (`serve.py`)

`serve.py` is the read side. It never writes the repo — it serves it. Two MCP
tools:

- **`query_knowledge(question, topic?)`** — two-stage KB retrieval over the
  `topics/` cards: BM25 pulls a wide candidate set (recall), then a local
  FlashRank cross-encoder reranks it for precision (see *Search rerank* below).
  Free, fast, deterministic; no web, no LLM. Returns ranked dated cards + a
  coverage report.
- **`research(question)`** — tiered grounded research. Gathers from three
  tiers, in priority order, then synthesizes:
  1. **KB** — BM25 over the `topics/` cards (the already-sifted cache).
  2. **Curated sources** — the arXiv query API (terms derived from the
     question) + domain-scoped search of our curated source set (`site:metr.org
     <q>` …). The trusted moat — *not* the open web.
  3. **Open web** — tuned DuckDuckGo: LLM-expanded queries, 2026+ floor,
     content-farm domains dropped. Supplementary, lower-trust, labelled.

  Two LLM calls: question → search plan (`prompts/expand.md`), then synthesis
  (`prompts/research.md`). Returns a JSON grounding contract — `answer`,
  `citations` (url/title/date/**tier**), `confidence`, `retrieval_date`,
  `warnings`, `sources_used`. Every claim must trace to a retrieved source.

The curated-domain set is derived from the watcher's own source registry
(`sources.py` / `sources_research.py` — the `domain` field, `curated_domains()`),
so it stays in sync. Web search works with **no API keys** (ddgs); set
`BRAVE_API_KEY` / `TAVILY_API_KEY` for higher-quality results. KB retrieval uses
BM25, not embeddings — no vector DB, no index to rebuild.

**Search rerank (`rerank.py`).** BM25 is recall-only: it matches keywords, so a
paraphrased question ("how do I make my agents reliable") misses a card titled
"Reducing tool-call errors in agents". `search_kb` (`query.py`) therefore pulls
a wide BM25 candidate set (`RERANK_CANDIDATES`, default 50) and reranks it with a
**FlashRank cross-encoder** (`ms-marco-MiniLM-L-12-v2`, ~34 MB, local CPU,
Apache-2 — no vector DB, no API spend), returning the reranked top-N. The
FlashRank call is a single seam (`rerank._cross_encoder_scores`), stubbed in
tests. It degrades gracefully: if the model can't load or `RERANK_DISABLED=1`,
retrieval falls back to plain BM25 order, so the cron path (which never searches)
and the test suite never depend on the model download. The model is downloaded
on first use to `FLASHRANK_CACHE` (a writable bind-mount on the VPS so it
persists across restarts).

**Knowledge graph (`/k/graph`).** A public, framework-free interactive graph of
the KB: nodes are cards (colored by topic, sized by rank), edges are the
`links.json` related-card links. Built by the pure `graph.build_graph_payload`
(`graph.py`) and written as `k/graph.json` by `synthesize.py` alongside the
magazine (best-effort, never aborts synthesis); rendered client-side by
`serve.py` with a vendored Cytoscape.js (`vendor/cytoscape.min.js`, no CDN). Tap
a node for a side panel (source / authority / relevance / summary, "Open source",
"Ask about this" deep-link into `/k/ask`, connected cards); topic-filter chips +
a search box highlight nodes.

**Research-trends dashboard (`/k/trends`).** A framework-free "Demand × Coverage"
view of the read-side demand signal: an SVG bubble scatter (x = how often a theme
is asked, y = how well the KB covers it) with the high-demand / low-coverage
**gap zone** shaded, a ranked theme bar list with `GAP` badges, and a
"most-demanded cards" strip. Themes + card demand come from `demand.py`
(`demand_themes` / `card_demand`), computed **live** on each request from
`query_log.jsonl` (pure Python — no LLM, no network, nothing written) and served
by `_trends_payload`. A 7/30/90-day window switch re-queries the same endpoint.
It is a plain `/k` surface like the graph — linked from the Magazine/Ask/Graph
navs and reachable at `https://example.com/k/trends`. serve.py applies no auth of
its own to `/k*`; whatever access layer fronts the deployment guards the whole
`/k` surface (and therefore the agent questions this page surfaces) uniformly.

## Run locally (dev)

No configuration is needed to see it work. The repository ships a small demo
corpus at `examples/seed-kb/` — notes about this engine itself — and that is the
default `KNOWLEDGE_DIR`, so a fresh clone answers questions out of the box:

```bash
uv sync --extra dev
uv run pytest -q                      # full suite, no network, no API key
uv run python serve.py --help
MCP_TRANSPORT=stdio uv run python serve.py   # MCP server over the seed corpus
```

Only the paths that call an LLM (`--selftest`, the watchers, `synthesize.py`)
need `OPENROUTER_API_KEY`; `query_knowledge` over the seed corpus does not.

```bash
cp .env.example .env          # OPENROUTER_API_KEY and the messaging transport

uv run python watch.py --dry-run           # mentors: prints would-be entries
uv run python research.py --dry-run        # research: prints sifted entries
uv run python research.py --seed           # populate research_state.json only
uv run python synthesize.py --dry-run      # synthesis: prints would-be topic pages
uv run python synthesize.py --backfill     # (re)build all topic pages from inbox

uv run python serve.py --selftest "<question>"   # run research once, print JSON, exit
MCP_TRANSPORT=stdio uv run python serve.py       # MCP server over stdio (local Claude Code)
uv run python serve.py                           # MCP server over httpStream (deploy mode)
```

`watch.py` / `research.py` support `--dry-run` (no write/DM/persist) and
`--seed` (populate state, no sift). `synthesize.py` supports `--dry-run`,
`--backfill` (reprocess the whole inbox) and `--rerank-all`. A `--dry-run`
still calls the LLM, so it needs a real `OPENROUTER_API_KEY`; `--seed` does not.

## Deploy

The engine is a plain Docker image plus a scheduler. There are four entrypoints:
three one-shot cron jobs (`watch.py`, `research.py`, `promote.py`, each followed
by `synthesize.py`) and one long-lived server (`serve.py`).

Deployment is deliberately not vendored into this repository — a compose file or
a Kubernetes manifest encodes one site's hostnames, mounts and routing, and is
misleading anywhere else. See **[docs/deploying.md](docs/deploying.md)** for the
shape the engine needs: the two volumes, the environment, the scheduling order,
and the auth surface you must put in front of it.

The one ordering rule that is not obvious: `synthesize.py` runs **last in every
chain**, after the watchers have appended to `inbox/`, so a single commit
captures the raw entries *and* the regenerated topic pages, and the digest fires
once per chain rather than once per watcher.

**`promote.py` — the enrichment loop.** It runs between `research.py` and
`synthesize.py` in the research chain. It mines the deep-research **history** the
long-lived `serve.py` persists (`$RESEARCH_HISTORY_DIR/*.json`) and promotes the *sources* those
interactive runs discovered — never the synthesized answer — into
`references/<slug>.md`. Each candidate URL is re-fetched, LLM-sifted for
relevance (dropped below `PROMOTE_REL_BAR`, default 3) and authority-tagged, so a
low-trust page is visibly marked, never silently trusted. `query.load_cards`
then indexes `references/` for `query_knowledge` / `research`, off the magazine
and graph. Like the watchers, `promote.py` **never runs git** (the host cron
commits) and saves its processed-id state (`promote_state.json`) only after a
successful pass. Demand (below) promotes high-demand / under-covered sources
first. Dry-run: `python promote.py --dry-run`.

**Demand signal (`demand.py`).** `synthesize.py` reads the query log
(`$QUERY_LOG_FILE`) and attaches a per-card *demand* score — how often
recent agent queries retrieve each card (recency-weighted BM25 replay). It blends
into the rank via `weights.demand` in `ranking.toml` (0 by default, so a no-op
until tuned; an empty log is graceful). `scripts/query_trends.py` prints the
"Google-Trends" view: recurring question themes, frequency, coverage, and a GAP
flag for high-demand / low-coverage themes — the enrichment priority list.

## Environment variables

| Var | Required | Default | Notes |
|---|---|---|---|
| `MCP_URL` | no | `http://whatsapp-mcp:39001/mcp` | Use `https://mcp.example.com/mcp` off-network. |
| `MCP_AUTH_TOKEN` | yes | — | Same token as the whatsapp-mcp stack. |
| `RECIPIENT_JID` | yes | — | Recipient id in `<number>@<jid-domain>` form. |
| `OPENROUTER_API_KEY` | yes | — | From openrouter.ai. The **only** provider key — chat and Whisper both. |
| `LLM_MODEL` | no | `deepseek/deepseek-chat` | Any OpenRouter chat model. Per-seam overrides: `LINK_MODEL`, `SYNTH_MODEL`, `LONG_SUMMARY_MODEL`. |
| `WHISPER_MODEL` | no | `openai/whisper-large-v3` | Tier-B video transcripts. OpenRouter has no `-turbo` build. |
| `SYNTH_MODEL` | no | = `LLM_MODEL` | Model for `synthesize.py` classification. |
| `KNOWLEDGE_DIR` | no | `examples/seed-kb/` | The corpus root. Unset, the read path falls back to the bundled seed corpus so a fresh clone runs; the write paths (`watch.py`, `ingest_oneshot.py`) require it explicitly. |
| `MAGAZINE_URL` | no | `https://example.com/k` | Where the pointer DM sends you to read/search — the magazine. |
| `MAX_ITEMS_PER_SOURCE` | no | `10` | Latest N items pulled per source per run. |
| `RESEARCH_MCP_TOKEN` | server | — | `serve.py` bearer token. Unset → server is unauthenticated (warns). |
| `MCP_HOST` / `MCP_PORT` | no | `0.0.0.0` / `39010` | `serve.py` httpStream bind. |
| `MCP_TRANSPORT` | no | `http` | `serve.py` transport — `http` or `stdio`. |
| `BRAVE_API_KEY` | no | — | `serve.py` web search, primary backend. |
| `TAVILY_API_KEY` | no | — | `serve.py` web search, fallback. ddgs (keyless) is the last resort. |
| `RERANK_DISABLED` | no | — | Set `1` to bypass FlashRank and keep raw BM25 order. |
| `RERANK_MODEL` | no | `ms-marco-MiniLM-L-12-v2` | FlashRank cross-encoder model name. |
| `RERANK_CANDIDATES` | no | `50` | BM25 candidate pool size handed to the reranker. |
| `FLASHRANK_CACHE` | no | (lib default) | Writable dir for the downloaded rerank model; set to a bind-mount so it persists. |
| `GRAPH_URL` | no | `/k/graph` | Graph link target embedded in the magazine nav. |

## Adding a source

- **Mentor** (trusted person) → append to `SOURCES` in `sources.py`.
- **Research feed** (needs date-filter + sift) → append to `RESEARCH_SOURCES`
  in `sources_research.py` with `"kind": "rss"` (or `"arxiv"`).

The `name` is the state-tracking key — don't rename existing sources casually
or all their items look new on the next run.

## Layout

```
knowledge-engine/
├── watch.py              # mentor entrypoint + shared helpers (state, feed, summarize)
├── research.py           # research-feed entrypoint (date-filter, arXiv retry, sift)
├── synthesize.py         # topic-synthesis entrypoint (inbox/ → topics/)
├── serve.py              # research MCP server (4th entrypoint — long-lived)
├── query.py              # research logic: BM25 search_kb (+ FlashRank rerank) + research() loop
├── rerank.py             # FlashRank cross-encoder rerank seam (precision over BM25 recall)
├── graph.py              # pure build_graph_payload → k/graph.json (nodes=cards, edges=links)
├── magazine.py           # pure render of the static magazine (k/index.html + k/cards.json)
├── search_backends.py    # pluggable web search (Brave/Tavily/ddgs) + URL fetch
├── knowledge.py          # repository helpers: sift/inbox/card parsing, entry + page render
├── ranking.py            # pure rank math (recency decay, weighted sum, sort)
├── topics.py             # topic registry + source→authority lookup
├── vendor/               # vendored static assets (cytoscape.min.js — no CDN)
├── sources.py            # mentor feeds
├── sources_research.py   # research feeds + arXiv query builder
├── backfill.py           # one-off historical seeder
├── prompts/              # sift_mentor.md, sift_research.md, classify.md, summarize.md, research.md
├── tests/                # pytest unit tests (pure helpers only)
├── Dockerfile / pyproject.toml / .env.example
├── examples/seed-kb/     # bundled demo corpus — the default KNOWLEDGE_DIR
├── docs/deploying.md     # what a deployment needs (no site-specific manifest)
└── data/                 # bind-mount target; state files live here
```

## About this public export

This is the real repository with its history kept, filtered rather than squashed.
Some things are deliberately absent:

- **The knowledge corpus itself is not published.** `KNOWLEDGE_DIR` normally
  points at a private repository of accumulated notes. What ships instead is
  `examples/seed-kb/` — a small demo corpus of notes about this engine — and it
  is the default, so a fresh clone answers questions with no configuration.
- **No deployment manifest.** A compose file or a Kubernetes manifest encodes one
  site's hostnames, mounts and routing and is misleading anywhere else.
  [`docs/deploying.md`](docs/deploying.md) describes the shape a deployment needs
  — the two volumes, the environment, the scheduling order, the auth surface you
  must put in front of it — without pretending to be that site.
- **No credentials, and no host names.** Secrets are referenced by variable name
  only; `.env.example` is a template. Hostnames in the docs and defaults are
  `example.com` placeholders.
- **The messaging recipient** is a configuration value (`RECIPIENT_JID`), never a
  checked-in number.

```bash
uv sync --extra dev
uv run pytest -q     # 366 tests, no network, no API key (re-derived 2026-09-17)
```

## Design notes (in case you forget)

- **Why entrypoints, not a framework?** `watch.py` / `research.py` /
  `synthesize.py` / `serve.py` / `backfill.py` are plain scripts sharing helpers
  by import — same pattern, no harness. Frameworks earn their cost at agent #3+.
  See the `feedback-no-framework-before-three` memory. `serve.py` uses the `mcp`
  SDK as a *server*; that is a protocol library (same SDK the watcher already
  uses as a WhatsApp client), not an agent framework.
- **Why BM25, not embeddings, for `research`?** ~100 markdown cards of
  technical-entity queries is the textbook BM25-wins case; embeddings would add
  a vector index to rebuild on every card change for no real gain. The
  knowledge repo's own `PageIndex` card says the same thing.
- **Why does synthesis classify-only, not LLM-rewrite the page?** A full-page
  LLM rewrite is non-deterministic, untestable under the no-mock TDD rule, and
  can silently drop or distort cards. The LLM does one thing Python can't —
  semantic classification — and Python does everything mechanical. Topic pages
  are a reproducible render; git diffs stay clean.
- **Why does the watcher not touch git?** It only writes files to the
  `/knowledge` bind-mount; the host cron does the commit+push, reusing the
  VPS's existing SSH deploy key. No credentials inside the container.
- **Why OpenRouter and not the Anthropic API?** Andre refuses paid Anthropic
  API for personal work. He consolidated every provider onto OpenRouter on
  2026-08-24 and closed the Groq and DeepInfra accounts — the pain was
  holding a separate prepaid balance with each vendor, not the cost.
  DeepSeek-V3 (`deepseek/deepseek-chat`) still sifts well and is unchanged
  by the move; OpenRouter resells the same weights.
- **Why bind-mount, not docker volume?** Direct `cat` access on the VPS, and
  Borg picks up `/storage/*` automatically.
- **Why `build_message` is still here** (`watch.py`): `backfill.py` — the
  one-off historical seeder — still builds digest-style messages. The live
  paths use `knowledge.format_entry` + `pointer_message` instead.

## License

MIT — see [`LICENSE`](LICENSE).