Knowledge Agent Server

About

独立的分层记忆后端，给 Knowledge Agent 用：一份代码同时服务 VS Code 扩展（Context Recall）、DSH 知识面板、任意 MCP 宿主（DSH / Claude Desktop / Cursor）和内置只读图谱浏览器。它把「证据」留在原处，只在检索时把词法、CJK 逐字索引、256 维向量和图谱关系四条通道合并成一个带关系的排序结果——导入是幂等增量的，没有任何删除路径。

• 只读优先：除三个 /v1/import/* 和 relink 外，所有端点只读；MCP 进程有路径白名单，导入与 relink 从该进程不可达。

• 可解释：融合排序用 RRF，每条结果能回溯到通道、词重叠与图边；跨文档关系来自共享实体，不来自模型猜测。

• 无模型依赖：服务从不调用大模型；语义抽取由调用方（扩展/宿主）提供，确定性实体链接始终可用。

Current version 0.1.8 (2026-11-04). The version is defined once in src/knowledge_agent_service/__init__.py; pyproject.toml, /health, /v1/capabilities and the MCP server info are kept equal to it by tests/test_version.py. Per-release history, commit ranges and known version debt live in CHANGELOG.md — this file only describes the code as it is today.

Related MCP server: rag-mcp-server

Architecture

flowchart TB
  subgraph clients["调用方"]
    ext["VS Code 扩展<br/>Context Recall"]
    panel["DSH 知识面板<br/>3D 图谱 / 检索"]
    mcp["MCP 宿主<br/>DSH · Claude · Cursor"]
    web["图谱浏览器<br/>/graph/"]
  end

  subgraph api["knowledge-agent-server :8765"]
    http["FastAPI 只读端点<br/>/v1/search · /v1/recall · /v1/relations"]
    graphapi["图谱端点<br/>/v1/graph/*"]
    write["写端点（唯一入口）<br/>/v1/import/* · /v1/graph/relink"]
    link["确定性实体链接<br/>entity_linker.py"]
    fuse["RRF 融合排序<br/>semantic 1.0 · lexical 1.2 · graph 0.8"]
  end

  subgraph stores["存储（全部本地）"]
    memory[("memory.sqlite3<br/>raw / candidate / memory<br/>FTS5 + CJK 逐字索引")]
    vectors[("vectors.sqlite3<br/>256 维 cosine")]
    neo[("Neo4j<br/>KnowledgeEntity / Document<br/>additive MERGE")]
  end

  ext -->|HTTP| http
  panel -->|HTTP| http
  panel -->|HTTP| graphapi
  web -->|HTTP| graphapi
  mcp -->|stdio JSON-RPC| http
  http --> fuse --> memory
  fuse --> vectors
  fuse --> neo
  graphapi --> neo
  write --> link --> neo
  write --> memory
  write --> vectors

分层的理由见 Layers：证据不动，SQLite 记生命周期，向量库独立，Neo4j 只增不删，排序在服务端只做一次。

Screenshots

全部截图来自本机实例（2026-11-11，1440×900），不是设计稿。分三组：

DSH 知识面板 · dock 抽屉（右下角「知识控制台」，6 个标签）

DSH 知识面板 · 3D 大屏（/knowledge-panel/view）

内置只读图谱浏览器 /graph/（实体视图，筛选与图例在左栏）

截图里只对内部文档编号与项目名做了模糊（GemMountain、GNR_EDS、639866、776846）；Birch Stream、Whitley、EGS 这类公开的平台代号保持原样。模糊在渲染时按标签当前文本实时施加，不是事后涂改。

dock 两张是在本地 mock 宿主页上渲染的——面板需要 DSH 的 /config、/status、/search 等宿主路由，浏览器无法直接打开；数据仍来自本机真实服务，未改动返回内容。

Naming

One canonical name: knowledge-agent-server.

以下标识故意保留 knowledge-agent-service 旧拼写，因为改名会移动数据或打断正在运行的部署，必须与迁移一起做：

• ~/.local/share/knowledge-agent-service — 数据目录（线上实例约 5 万条记忆）；可用 KNOWLEDGE_AGENT_DATA_DIR 覆盖，迁移时改指向即可。

• ~/.config/knowledge-agent-service/environment — 运行时配置。

• knowledge-agent-service.service — 已安装的 systemd 用户单元（以及写它的 scripts/install-user-service.sh）。

• Python 导入包 knowledge_agent_service 与本地检出目录 ~/knowledge-agent-service。

API Endpoints

Every route the service serves. Only POST /v1/import/* and POST /v1/graph/relink write anything; the rest are read-only by construction.

Layers

1. Evidence remains immutable at its source and is referenced by ID.

2. SQLite stores scoped raw, candidate, and active memory records with FTS5 and audit events.

3. A separate SQLite vector store persists embeddings and provides semantic recall.

4. Neo4j stores normalized knowledge entities and relationships through additive MERGE operations only.

5. The API combines lexical and vector recall while preserving evidence and lifecycle state.

Document Graph

Imported PDFs do not stop at Document/DocumentChunk nodes. Two layers connect them to the knowledge graph:

1. Deterministic entity linking (always on). entity_linker.py matches chunk text against the alias catalogue of active KnowledgeEntity nodes — label first, then explicit aliases, then tags. Matching is NFKC-normalized and casefolded, enforces word boundaries for ASCII aliases, allows substring matches for CJK aliases, gives the longest surface form priority, and resolves one alias pointing at several entities deterministically. It needs no model, so the result is reproducible and cheap enough to run on every import. Every match becomes a MENTIONS relationship:

   (:DocumentChunk)-[:MENTIONS {alias, source, occurrences, confidence}]->(:KnowledgeEntity)
   (:Document)-[:MENTIONS {alias, chunks, occurrences, confidence, evidence_chunk_ids}]->(:KnowledgeEntity)

   Without this layer an imported document is an isolated subgraph. Two documents mentioning the same entity are related as soon as both carry MENTIONS edges to it.

2. Semantic extraction (caller supplied). The service never calls a model. The VS Code extension reads document excerpts with the active chat model and posts the resulting entities, aliases, and relationships through the existing snapshot sync path, which reaches Neo4j through the same additive upsert.

Because linking depends on the catalogue, POST /v1/graph/relink rebuilds MENTIONS edges for already stored chunks after the catalogue grows — no re-import needed. POST /v1/graph/document-links returns cross-document pairs derived from shared entities, and /v1/search appends the highest-scoring pairs to the graph array so retrieval sees them without a client change. Each pair carries the shared entity IDs and the source chunk evidence on both sides. The document-links response always carries links and count, so a skipped or failed lookup is shaped exactly like an empty result.

Link Quality And Safety

include_tags defaults to false. Tags are topical labels attached to sessions and decisions — hardware, implementation, git — not entity names. On a four-document corpus of 2,869 chunks the difference was measured, not assumed:

The tag-derived links were overwhelmingly generic words such as board, register, status, and platform that match nearly every datasheet page, and they added almost no distinct entities. File- and symbol-like surfaces (.github/agents/yocto.agent.md, tasks.json) are rejected for the same reason.

max_alias_document_frequency is available but off by default. Setting it to 0.25 drops aliases present in more than a quarter of chunks, which cuts mentions roughly in half by removing ubiquitous tokens such as Intel — useful when the goal is ranking document pairs by distinctiveness rather than collecting chunk-level evidence.

POST /v1/graph/relink defaults to dry_run: true because Neo4j writes here are additive and the adapter exposes no delete operation. A dry run reports exactly what would be written, including planned_links, a preview of the cross-document pairs the run would create. Pass dry_run: false to persist.

A committed relink is still reversible by hand, since every generated relationship is namespaced:

MATCH ()-[r:MENTIONS]->() WHERE r.id STARTS WITH 'mention:' DELETE r

POST /v1/import/continuity accepts Continuity decisions plus optional embeddings and imports them additively. Repeating an import updates the same decision IDs without deleting existing data. The health response reports the number of records imported from Continuity.

The default data directory is ~/.local/share/knowledge-agent-service. The service never depends on an OpenBMC checkout.

Retrieval With Relationships

POST /v1/search returns the three raw channels and leaves ranking to the caller. POST /v1/recall does the ranking once, on the service side, and returns the edges between the winners:

• results — fused ranking (semantic + lexical + graph, reciprocal-rank fusion with query-term overlap)

• relationships — graph edges (source: graph), cross-document links (document-link), and results sharing evidence (shared-evidence)

• links / entities — the raw graph rows the relationships were built from

• meta.counts — per-channel counts, so a caller can tell "no relations" from "graph unavailable"

The endpoint is read-only. It only reads what /v1/search already fetches; it never calls /v1/import/* or /v1/graph/relink.

curl -s -X POST -H 'content-type: application/json' \
  -d '{"query":"eSPI","limit":5}' \
  http://127.0.0.1:8765/v1/recall

POST /v1/relations answers the narrower "how are these things connected" question. It resolves the English tokens of the question to known entities and returns their one-hop RUNS_ON / RELATED edges as relation cards; whether it prefers outgoing edges is decided from the wording (跑在, 支持哪些, 依赖什么, …), and no recognised entity yields matched: false so the caller can fall back to /v1/recall. It is read-only and does not search memory.

Document Metadata Filters

/v1/search — and therefore /v1/recall, which extends the same request body — accepts five optional filters: collection, exclude_collection, doc_type, model and doc_ids.

• Matching is exact membership against the document tags stored in tags_json, evaluated with SQLite's json_each, so it narrows the candidate pool in SQL: before ORDER BY/LIMIT, on all three channels (FTS, CJK, semantic).

• doc_ids is an OR over its values; collection / doc_type / model AND together; exclude_collection is the negation (NOT EXISTS). Empty strings and empty lists add no clause, so an unfiltered request runs exactly the SQL and returns exactly the field set it did before the filters existed.

• Rows that carry document metadata gain a doc_metadata key (collection / doc_type / doc_id / version / lang / source_url). Plain memories keep their previous field set.

• The same keys are written onto chunks by POST /v1/import/documents; empty values are omitted rather than stored as empty strings.

MCP Server

The repository ships a read-only stdio MCP server, so hosts that speak MCP (DSH, Claude Desktop, Cursor) can use the same capability without the VS Code extension:

PYTHONPATH=src knowledge-agent-mcp
# or
PYTHONPATH=src python3 -m knowledge_agent_service.mcp_server

Every request goes through a path allow-list (READ_ONLY_PATHS / READ_ONLY_PREFIXES in mcp_server.py); import and relink endpoints are unreachable from this process. KNOWLEDGE_SERVICE_URL sets the service address (default http://127.0.0.1:8765), KNOWLEDGE_MCP_TIMEOUT the timeout.

knowledge_recall computes the same 256-dimension query embedding as the VS Code extension (embedding.py, pinned by tests/test_embedding.py against vectors generated by the JavaScript implementation), so the semantic channel keeps working for MCP callers.

Example DSH registration:

- id: mcp-knowledge-server
  name: '@deepseek-ai/dsh-mcp-client'
  config:
    serverName: knowledge
    transport: stdio
    command: /usr/bin/python3
    args: ['-m', 'knowledge_agent_service.mcp_server']
    cwd: /home/xuqj/knowledge-agent-service
    env:
      PYTHONPATH: /home/xuqj/knowledge-agent-service/src
      KNOWLEDGE_SERVICE_URL: http://127.0.0.1:8765

Run

python3 -m venv .venv
.venv/bin/pip install -e '.[dev]'
.venv/bin/knowledge-agent-server serve

The API listens on 127.0.0.1:8765 by default. Override it with KNOWLEDGE_AGENT_HOST and KNOWLEDGE_AGENT_PORT.

GET /health and GET /v1/capabilities both report version and commit for the code actually serving requests, so a stale process is visible instead of silently answering with old behaviour.

CLI

knowledge-agent-server serve                    # run the HTTP service
knowledge-agent-server migrate-extension <snapshot> [--apply] [--sync-neo4j]
knowledge-agent-server reindex-cjk [--dry-run] [--batch-size 500]

knowledge-agent-service still works as a deprecated alias of the same entry point. reindex-cjk rebuilds the CJK character index that gives Chinese queries substring recall; --dry-run reports what would be written. migrate-legacy-memory exists but is internal: only scripts/legacy-db-guard.py apply may invoke it, against a guard-staged read-only snapshot.

Tests

# Write nothing outside an isolated directory (see the acceptance-isolation rule below).
export KNOWLEDGE_AGENT_DATA_DIR=/tmp/knowledge-agent-tests-$(date +%Y%m%d%H%M%S)
PYTHONPATH=src python3 -m unittest discover -s tests

241 tests, standard-library unittest (no pytest required; .[dev] installs it only for convenience). api.py builds its stores at import time, so a writable KNOWLEDGE_AGENT_DATA_DIR is mandatory — without it the import fails with unable to open database file instead of running against a half-open store.

tests/test_version.py fails if __init__.__version__, pyproject.toml, SERVICE_VERSION and the MCP server info ever disagree.

Start At Boot

Install and immediately start the user-level systemd service:

./scripts/install-user-service.sh

The unit name stays knowledge-agent-service.service (see Naming); renaming it needs a unit migration, not just a file rename. The installer keeps Python dependencies in this project's .runtime directory, writes the unit to ~/.config/systemd/user/knowledge-agent-service.service, and enables it for the user manager. With systemd user lingering enabled, it starts during machine boot without an interactive login.

Runtime settings are read from ~/.config/knowledge-agent-service/environment. The file is created with mode 0600; Neo4j remains disabled unless its settings are explicitly added. Starting the service never imports data or synchronizes Neo4j.

Useful operations:

systemctl --user status knowledge-agent-service
systemctl --user restart knowledge-agent-service
journalctl --user -u knowledge-agent-service -f
systemctl --user disable --now knowledge-agent-service

Neo4j

Set KNOWLEDGE_NEO4J_URI, KNOWLEDGE_NEO4J_USER, KNOWLEDGE_NEO4J_PASSWORD, and optionally KNOWLEDGE_NEO4J_DATABASE. Credentials are read only at runtime and must not be committed.

Neo4j writes are additive. The adapter contains no database-clearing operation and represents source relationship vocabulary as a RELATED relationship with a type property.

Non-Destructive Migration

Inspect an existing extension snapshot without writing:

knowledge-agent-service migrate-extension /path/to/knowledge-agent.json

Apply an incremental import into the service's own databases:

knowledge-agent-service migrate-extension /path/to/knowledge-agent.json --apply

Add --sync-neo4j only when Neo4j credentials are configured. The migration verifies that the source file hash is unchanged and never deletes source or target records.

Legacy Database Guard

Legacy Memory Agent SQLite databases must only be opened through scripts/legacy-db-guard.py, never with ad hoc Python or SQLite commands.

Inspect a real legacy database read-only (safe on the live file — it is opened with mode=ro&immutable=1, PRAGMA query_only=ON, and verified unchanged by SHA-256 before and after):

python3 scripts/legacy-db-guard.py inspect --source /path/to/copilot-kb/memory.sqlite3

The report contains only aggregate metadata (source path, SHA-256, size, row counts, status/memory_kind/loading_policy distributions, and evidence/source/workspace/repo coverage counts). It never prints stored summary, resolution, context, or evidence content.

apply is a safety proxy for the future legacy migration CLI. It refuses staging directories that overlap the source parent, the default production directory ~/.local/share/knowledge-agent-service, are non-empty, are symlinks, or contain the source. Before running anything it copies the source into the staging directory as a consistent, integrity-checked, read-only snapshot (legacy-source.sqlite3, produced with the SQLite backup API from a mode=ro&immutable=1 handle and then chmod 0444). It then runs the fixed command <python> -m knowledge_agent_service.cli migrate-legacy-memory <SNAPSHOT> --apply — using the guard's own interpreter (sys.executable), never a hardcoded python — with KNOWLEDGE_AGENT_DATA_DIR pointed at the staging directory. The importer never receives the real source path. The real source is SHA-256 verified unchanged before and after, including on importer failure or exception:

python3 scripts/legacy-db-guard.py apply --source /path/to/copilot-kb/memory.sqlite3 --staging-dir /tmp/legacy-staging

apply then runs the fixed internal command <python> -m knowledge_agent_service.cli migrate-legacy-memory <SNAPSHOT> --apply with KNOWLEDGE_AGENT_DATA_DIR pointed at the staging directory. The importer is an internal, guard-controlled command and is never run directly by normal users: it only accepts a guard-staged, read-only legacy-source.sqlite3 snapshot located inside the staging data directory, and it refuses the production data directory. Do not run apply against a real database as a migration; a committed migration must be reviewed by a human first.

The legacy importer is conservative and non-destructive. It imports legacy memory additively and idempotently into the SQLite memory store only:

• Legacy active memory and all other non-raw/non-rejected records become status=draft, memory_kind=candidate, so they are never returned by default active retrieval.

• Legacy memory_kind=raw stays raw, and legacy status=rejected becomes rejected.

• Session contexts become type=session records with a fixed, content-free summary and the canonical context JSON preserved in resolution.

• Evidence, source refs, scope, timestamps, and provenance (imported_from, source_id, source_status, source_memory_kind, loading_policy, storage_targets, last_accessed_at) are preserved. loading_policy is never treated as proof of Host injection.

• The importer never writes vectors or Neo4j, never syncs Neo4j, and never generates embeddings.

To inspect a legacy database, use inspect. To migrate one, use apply against a safe, empty staging directory. The internal importer is only ever invoked by the guard with a fixed argument list; it accepts no Neo4j flags.

验收/试验数据隔离（强制）

任何会写库的验收、试验或迁移（CLI、脚本、扩展导入、/v1/import/*）都必须在写入前把数据目录切到独立目录，禁止复用生产库：

export KNOWLEDGE_AGENT_DATA_DIR=/tmp/knowledge-agent-acceptance-$(date +%Y%m%d%H%M%S)

KNOWLEDGE_AGENT_DATA_DIR 由 config.py:36-38 读取，Settings.memory_db / vector_db / event_log 都由它派生（config.py:18-28）。对服务进程要显式设置（systemd 单元的 Environment=/EnvironmentFile=，或 systemctl --user set-environment KNOWLEDGE_AGENT_DATA_DIR=... 后重启服务）；验收结束即删除该目录，绝不把默认生产目录 ~/.local/share/knowledge-agent-service 当作 staging。

教训：2026-09-11 与 2026-09-16 两批「面包/烤面包」验收测试直接写进了生产记忆库，留下 16 个 active 的 knowledge:* 单元并污染语义检索（审计结论见 knowledge:d36d1b9bff6138116732）。

巡检与清理：

# 只读自检：命中任何残留即 exit 1（可接 CI / 日常巡检），--json 落盘结构化结果
python3 scripts/check-test-residues.py
python3 scripts/check-test-residues.py --db /tmp/copy.sqlite3 --json /tmp/residues.json

# 清理：默认 dry-run；--backup 先做 SQLite backup API 备份；--apply 执行
python3 scripts/cleanup-test-residues.py
python3 scripts/cleanup-test-residues.py --apply --backup
python3 scripts/cleanup-test-residues.py --apply --confirm-production   # 生产库必须二次确认

检测规则：status='active' AND id LIKE 'knowledge:%' AND (summary LIKE '%面包%' OR id IN <内置 16-id 清单>)；resolution 不参与匹配，显式豁免集为空（保证 09-14 审计结论 knowledge:d36d1b9bff6138116732 不被误伤）。清理只做 active -> superseded 状态转换并写入 memory_events 审计（event_type='supersede'），不物理删除 SQLite 行、不改其他字段、不动 vectors.sqlite3；Neo4j 的 DETACH DELETE 只在带 --confirm-production 的生产 apply 执行。

图谱浏览器

服务内置只读图谱浏览器（替代已不可用的 Neo4j Browser）：静态 SPA 由服务本体挂载，前端产物随 Python 包发布，无 CDN / 外网请求，无新增 Python 依赖。

• 入口：http://<host>:8765/graph/；GET / 会 307 重定向到 /graph/（VS Code 端口转发后用 http://localhost:8765/graph/ 亦可）。

• 只读声明：全部端点只读，不写 Neo4j / SQLite / events.jsonl；新增 Cypher 不含 MERGE/CREATE/SET/DELETE/DROP，用户输入一律 $param 参数化，扩展逐跳查询且有界（每节点邻居 LIMIT ≤ 200，单请求查询次数 ≤ (depth+1)*4+2）。

• 失败降级：一律 HTTP 200 + error 字段 + 空形状；未配置 Neo4j 时返回 {"error": "neo4j-not-configured", ...空形状}，静态页面仍可加载。

端点（前缀 /v1/graph/，与 relink、document-links 同级）：

前端资源位于 src/knowledge_agent_service/static/graph/（构建与维护说明见该目录 README.md）；目录缺失时 /graph/ 返回 404，服务启动不受影响。

Releases

版本记录的唯一来源是 CHANGELOG.md；每个 tag 对应一个 GitHub Release。

发布流程见 CHANGELOG 的「发布约定」：改 __init__.__version__ → 补 CHANGELOG → 打 annotated tag → 推送（推 tag 需显式确认）。

Topics

仓库话题（GitHub 侧已设置，gh repo edit --add-topic）：

openbmc · bmc · knowledge-graph · retrieval · rag · embeddings · memory · neo4j · sqlite · fastapi · mcp · python

License

Apache License 2.0 — 允许商用、修改、分发，附带专利授权；分发时需保留版权与许可声明，并标明修改。

pyproject.toml 的 license 字段指向同一份 LICENSE，GitHub 侧会自动识别为 Apache-2.0。第三方前端依赖（sigma / graphology）的许可见 src/knowledge_agent_service/static/graph/vendor/licenses/。