amiticia-knowledge-engine

Watches AI/dev content and writes it into the AmiticIA knowledge repository (amiticia-knowledge), then DMs a pointer via the existing whatsapp-mcp service. Four entrypoints share the codebase — three one-shot cron jobs that write the repo, and one long-lived server that reads it:

• watch.py — mentors. Curated people we follow (Akita, Lucas Montano, Simon Willison). High trust: every item is summarized and scored, no relevance gate. Mon/Wed/Fri cron.

• research.py — research feeds. High-volume, noisy sources (arXiv, Import AI, Ahead of AI, AlphaSignal, METR, Epoch). Hard-filtered to 2026+ and LLM-sifted for relevance before anything is kept. Own cron.

• synthesize.py — topic synthesis. Reads the inbox/ the other two wrote, classifies each new entry into one of the fixed topics/, and rewrites those topic pages as rank-ordered card lists. Runs last in the cron chain. The LLM only classifies; all ranking and rendering is plain Python, so topic pages are reproducible and the LLM never rewrites accumulated knowledge.

• serve.py — research MCP server. A long-lived server (not a cron job) that exposes the knowledge repo as two MCP tools — query_knowledge (BM25 retrieval, free) and research (the grounded loop: KB + live web + LLM synthesis → a dated, cited answer). The tool to consult before an architecture decision, so the answer comes from current reality.

Why this exists: leading-edge AI-agent knowledge moves faster than any model's training data. This is durable external memory — current, dated, verified — so architecture decisions are made from 2026 reality, not stale training.

▶ Start here: serve.py — the read side, and the only long-lived process. Everything else exists to fill the corpus it serves.

Look at first

1. serve.py — the research MCP server. Two tools: query_knowledge (BM25 + a local cross-encoder rerank, free, deterministic, no network) and research (the grounded loop that returns a JSON contract — answer, citations with a tier per source, confidence, retrieval date, warnings). The tier label is the interesting part: a curated source and an open-web result are never presented as equally trustworthy.

2. prompts/ — every LLM call this system makes, as plain markdown files, one per seam: sift, classify, expand, link, research, summarize. The design rule is visible here — the LLM only ever sifts or classifies; ranking, rendering and page assembly are plain Python, so topic pages are reproducible and no accumulated knowledge is ever LLM-rewritten.

3. promote.py — the enrichment loop, and the best example of the trust posture. It mines the deep-research history the server persists and promotes the sources those runs discovered — never the synthesized answer. Each candidate URL is re-fetched, sifted for relevance and authority-tagged, so a low-trust page is visibly marked rather than silently absorbed.

Related MCP server: pramana-mcp

What each run does

1. Pull the latest items per source (sources.py / sources_research.py).

2. Skip items already in the seen-ID state (state.json / research_state.json).

3. research.py only: drop items dated before 2026.

4. LLM-sift each new item with DeepSeek-V3 via OpenRouter:

   • mentors → prompts/sift_mentor.md (summary + relevance 1-5, never skipped),

   • research → prompts/sift_research.md (summary + relevance, or SKIP).

5. Append the keepers as markdown entries to the knowledge repo's inbox/<YYYY-MM-DD>.md (knowledge.py).

6. Persist seen IDs. (The watchers no longer DM — synthesize.py is the sole digest author; see below. They print feed-health warnings to the cron log.)

The knowledge repo itself lives at KNOWLEDGE_DIR; on the VPS that is a git clone, and the host cron commits + pushes after each run (see Deploy).

Synthesis (synthesize.py)

After the two watchers append to inbox/, synthesize.py rolls those entries into the readable topics/ pages:

1. Parse every inbox/*.md entry; skip those already in synth_state.json.

2. One LLM call per new entry → a topic slug + optional business "so what" + optional supersedes flag (prompts/classify.md).

3. For each touched topic: parse the existing cards out of the page, add the new ones (deduped by link), recompute every card's rank (relevance + recency-decay + authority, weights in the repo's ranking.toml), re-sort, rewrite topics/<slug>.md.

It only ever writes topics/, AGENTS-BRIEFING.md, the magazine under k/, links.json and synth_state.json — never inbox/, never git. --backfill reprocesses the whole inbox ignoring state; --rerank-all re-ranks every page without needing new entries (no LLM call).

After ranking, two things happen (both best-effort — a failure never aborts the canonical topic-page write):

• Related links. For each new/changed card, BM25 proposes up to eight keyword-similar candidates (links.py, reusing the query.py retrieval), and one bounded LLM call (prompts/link_related.md) keeps the 2-3 genuinely related ones with a short reason. These are sidecar metadata in links.json (keyed by card link, cached by content hash) — cards are never mutated, so the classify-not-rewrite rule holds. The topic pages and magazine render a "Related" block from them.

• Themed digest DM. synthesize.py is now the sole WhatsApp digest author: it groups this run's new cards by topic (📚 *Knowledge digest*, one *Topic* section each, • title — so-what lines, ≤8 with an "…and N more" tail), and rides the dead-feed + stale-topic ⚠️ warnings (read from both ingest stats files) on the same DM. Sent only when something new was classified, after which synth_state.json and links.json are saved — the after-send rule, so a failed send replays cleanly.

Research MCP server (serve.py)

serve.py is the read side. It never writes the repo — it serves it. Two MCP tools:

• query_knowledge(question, topic?) — two-stage KB retrieval over the topics/ cards: BM25 pulls a wide candidate set (recall), then a local FlashRank cross-encoder reranks it for precision (see Search rerank below). Free, fast, deterministic; no web, no LLM. Returns ranked dated cards + a coverage report.

• research(question) — tiered grounded research. Gathers from three tiers, in priority order, then synthesizes:

  1. KB — BM25 over the topics/ cards (the already-sifted cache).

  2. Curated sources — the arXiv query API (terms derived from the question) + domain-scoped search of our curated source set (site:metr.org <q> …). The trusted moat — not the open web.

  3. Open web — tuned DuckDuckGo: LLM-expanded queries, 2026+ floor, content-farm domains dropped. Supplementary, lower-trust, labelled.

  Two LLM calls: question → search plan (prompts/expand.md), then synthesis (prompts/research.md). Returns a JSON grounding contract — answer, citations (url/title/date/tier), confidence, retrieval_date, warnings, sources_used. Every claim must trace to a retrieved source.

The curated-domain set is derived from the watcher's own source registry (sources.py / sources_research.py — the domain field, curated_domains()), so it stays in sync. Web search works with no API keys (ddgs); set BRAVE_API_KEY / TAVILY_API_KEY for higher-quality results. KB retrieval uses BM25, not embeddings — no vector DB, no index to rebuild.

Search rerank (rerank.py). BM25 is recall-only: it matches keywords, so a paraphrased question ("how do I make my agents reliable") misses a card titled "Reducing tool-call errors in agents". search_kb (query.py) therefore pulls a wide BM25 candidate set (RERANK_CANDIDATES, default 50) and reranks it with a FlashRank cross-encoder (ms-marco-MiniLM-L-12-v2, ~34 MB, local CPU, Apache-2 — no vector DB, no API spend), returning the reranked top-N. The FlashRank call is a single seam (rerank._cross_encoder_scores), stubbed in tests. It degrades gracefully: if the model can't load or RERANK_DISABLED=1, retrieval falls back to plain BM25 order, so the cron path (which never searches) and the test suite never depend on the model download. The model is downloaded on first use to FLASHRANK_CACHE (a writable bind-mount on the VPS so it persists across restarts).

Knowledge graph (/k/graph). A public, framework-free interactive graph of the KB: nodes are cards (colored by topic, sized by rank), edges are the links.json related-card links. Built by the pure graph.build_graph_payload (graph.py) and written as k/graph.json by synthesize.py alongside the magazine (best-effort, never aborts synthesis); rendered client-side by serve.py with a vendored Cytoscape.js (vendor/cytoscape.min.js, no CDN). Tap a node for a side panel (source / authority / relevance / summary, "Open source", "Ask about this" deep-link into /k/ask, connected cards); topic-filter chips + a search box highlight nodes.

Research-trends dashboard (/k/trends). A framework-free "Demand × Coverage" view of the read-side demand signal: an SVG bubble scatter (x = how often a theme is asked, y = how well the KB covers it) with the high-demand / low-coverage gap zone shaded, a ranked theme bar list with GAP badges, and a "most-demanded cards" strip. Themes + card demand come from demand.py (demand_themes / card_demand), computed live on each request from query_log.jsonl (pure Python — no LLM, no network, nothing written) and served by _trends_payload. A 7/30/90-day window switch re-queries the same endpoint. It is a plain /k surface like the graph — linked from the Magazine/Ask/Graph navs and reachable at https://example.com/k/trends. serve.py applies no auth of its own to /k*; whatever access layer fronts the deployment guards the whole /k surface (and therefore the agent questions this page surfaces) uniformly.

Run locally (dev)

No configuration is needed to see it work. The repository ships a small demo corpus at examples/seed-kb/ — notes about this engine itself — and that is the default KNOWLEDGE_DIR, so a fresh clone answers questions out of the box:

uv sync --extra dev
uv run pytest -q                      # full suite, no network, no API key
uv run python serve.py --help
MCP_TRANSPORT=stdio uv run python serve.py   # MCP server over the seed corpus

Only the paths that call an LLM (--selftest, the watchers, synthesize.py) need OPENROUTER_API_KEY; query_knowledge over the seed corpus does not.

cp .env.example .env          # OPENROUTER_API_KEY and the messaging transport

uv run python watch.py --dry-run           # mentors: prints would-be entries
uv run python research.py --dry-run        # research: prints sifted entries
uv run python research.py --seed           # populate research_state.json only
uv run python synthesize.py --dry-run      # synthesis: prints would-be topic pages
uv run python synthesize.py --backfill     # (re)build all topic pages from inbox

uv run python serve.py --selftest "<question>"   # run research once, print JSON, exit
MCP_TRANSPORT=stdio uv run python serve.py       # MCP server over stdio (local Claude Code)
uv run python serve.py                           # MCP server over httpStream (deploy mode)

watch.py / research.py support --dry-run (no write/DM/persist) and --seed (populate state, no sift). synthesize.py supports --dry-run, --backfill (reprocess the whole inbox) and --rerank-all. A --dry-run still calls the LLM, so it needs a real OPENROUTER_API_KEY; --seed does not.

Deploy

The engine is a plain Docker image plus a scheduler. There are four entrypoints: three one-shot cron jobs (watch.py, research.py, promote.py, each followed by synthesize.py) and one long-lived server (serve.py).

Deployment is deliberately not vendored into this repository — a compose file or a Kubernetes manifest encodes one site's hostnames, mounts and routing, and is misleading anywhere else. See docs/deploying.md for the shape the engine needs: the two volumes, the environment, the scheduling order, and the auth surface you must put in front of it.

The one ordering rule that is not obvious: synthesize.py runs last in every chain, after the watchers have appended to inbox/, so a single commit captures the raw entries and the regenerated topic pages, and the digest fires once per chain rather than once per watcher.

promote.py — the enrichment loop. It runs between research.py and synthesize.py in the research chain. It mines the deep-research history the long-lived serve.py persists ($RESEARCH_HISTORY_DIR/*.json) and promotes the sources those interactive runs discovered — never the synthesized answer — into references/<slug>.md. Each candidate URL is re-fetched, LLM-sifted for relevance (dropped below PROMOTE_REL_BAR, default 3) and authority-tagged, so a low-trust page is visibly marked, never silently trusted. query.load_cards then indexes references/ for query_knowledge / research, off the magazine and graph. Like the watchers, promote.py never runs git (the host cron commits) and saves its processed-id state (promote_state.json) only after a successful pass. Demand (below) promotes high-demand / under-covered sources first. Dry-run: python promote.py --dry-run.

Demand signal (demand.py). synthesize.py reads the query log ($QUERY_LOG_FILE) and attaches a per-card demand score — how often recent agent queries retrieve each card (recency-weighted BM25 replay). It blends into the rank via weights.demand in ranking.toml (0 by default, so a no-op until tuned; an empty log is graceful). scripts/query_trends.py prints the "Google-Trends" view: recurring question themes, frequency, coverage, and a GAP flag for high-demand / low-coverage themes — the enrichment priority list.

Environment variables

Adding a source

• Mentor (trusted person) → append to SOURCES in sources.py.

• Research feed (needs date-filter + sift) → append to RESEARCH_SOURCES in sources_research.py with "kind": "rss" (or "arxiv").

The name is the state-tracking key — don't rename existing sources casually or all their items look new on the next run.

Layout

knowledge-engine/
├── watch.py              # mentor entrypoint + shared helpers (state, feed, summarize)
├── research.py           # research-feed entrypoint (date-filter, arXiv retry, sift)
├── synthesize.py         # topic-synthesis entrypoint (inbox/ → topics/)
├── serve.py              # research MCP server (4th entrypoint — long-lived)
├── query.py              # research logic: BM25 search_kb (+ FlashRank rerank) + research() loop
├── rerank.py             # FlashRank cross-encoder rerank seam (precision over BM25 recall)
├── graph.py              # pure build_graph_payload → k/graph.json (nodes=cards, edges=links)
├── magazine.py           # pure render of the static magazine (k/index.html + k/cards.json)
├── search_backends.py    # pluggable web search (Brave/Tavily/ddgs) + URL fetch
├── knowledge.py          # repository helpers: sift/inbox/card parsing, entry + page render
├── ranking.py            # pure rank math (recency decay, weighted sum, sort)
├── topics.py             # topic registry + source→authority lookup
├── vendor/               # vendored static assets (cytoscape.min.js — no CDN)
├── sources.py            # mentor feeds
├── sources_research.py   # research feeds + arXiv query builder
├── backfill.py           # one-off historical seeder
├── prompts/              # sift_mentor.md, sift_research.md, classify.md, summarize.md, research.md
├── tests/                # pytest unit tests (pure helpers only)
├── Dockerfile / pyproject.toml / .env.example
├── examples/seed-kb/     # bundled demo corpus — the default KNOWLEDGE_DIR
├── docs/deploying.md     # what a deployment needs (no site-specific manifest)
└── data/                 # bind-mount target; state files live here

About this public export

This is the real repository with its history kept, filtered rather than squashed. Some things are deliberately absent:

• The knowledge corpus itself is not published. KNOWLEDGE_DIR normally points at a private repository of accumulated notes. What ships instead is examples/seed-kb/ — a small demo corpus of notes about this engine — and it is the default, so a fresh clone answers questions with no configuration.

• No deployment manifest. A compose file or a Kubernetes manifest encodes one site's hostnames, mounts and routing and is misleading anywhere else. docs/deploying.md describes the shape a deployment needs — the two volumes, the environment, the scheduling order, the auth surface you must put in front of it — without pretending to be that site.

• No credentials, and no host names. Secrets are referenced by variable name only; .env.example is a template. Hostnames in the docs and defaults are example.com placeholders.

• The messaging recipient is a configuration value (RECIPIENT_JID), never a checked-in number.

uv sync --extra dev
uv run pytest -q     # 366 tests, no network, no API key (re-derived 2026-09-17)

Design notes (in case you forget)

• Why entrypoints, not a framework? watch.py / research.py / synthesize.py / serve.py / backfill.py are plain scripts sharing helpers by import — same pattern, no harness. Frameworks earn their cost at agent #3+. See the feedback-no-framework-before-three memory. serve.py uses the mcp SDK as a server; that is a protocol library (same SDK the watcher already uses as a WhatsApp client), not an agent framework.

• Why BM25, not embeddings, for research? ~100 markdown cards of technical-entity queries is the textbook BM25-wins case; embeddings would add a vector index to rebuild on every card change for no real gain. The knowledge repo's own PageIndex card says the same thing.

• Why does synthesis classify-only, not LLM-rewrite the page? A full-page LLM rewrite is non-deterministic, untestable under the no-mock TDD rule, and can silently drop or distort cards. The LLM does one thing Python can't — semantic classification — and Python does everything mechanical. Topic pages are a reproducible render; git diffs stay clean.

• Why does the watcher not touch git? It only writes files to the /knowledge bind-mount; the host cron does the commit+push, reusing the VPS's existing SSH deploy key. No credentials inside the container.

• Why OpenRouter and not the Anthropic API? Andre refuses paid Anthropic API for personal work. He consolidated every provider onto OpenRouter on 2026-08-24 and closed the Groq and DeepInfra accounts — the pain was holding a separate prepaid balance with each vendor, not the cost. DeepSeek-V3 (deepseek/deepseek-chat) still sifts well and is unchanged by the move; OpenRouter resells the same weights.

• Why bind-mount, not docker volume? Direct cat access on the VPS, and Borg picks up /storage/* automatically.

• Why build_message is still here (watch.py): backfill.py — the one-off historical seeder — still builds digest-style messages. The live paths use knowledge.format_entry + pointer_message instead.

License

MIT — see LICENSE.