ceph-doc-kb

Version-aware, component-scoped Ceph documentation knowledge base. Indexes upstream RST and IBM Storage Ceph HTML into per-component FAISS indices. Search is two-tier: BM25 keywords + fastembed semantic, then quality re-rank.

Use this MCP for how-to, architecture, IBM-only procedures, and copy-paste examples. Use ceph-cmd-kb to verify that a command in those docs is still valid for the target release.

For agents (read this first)

This KB contains multiple versions. If the user did not name one, ask before searching. list_versions() lists what is loaded.

IBM docs cover content that is not in upstream: container registry, licensing, crossgrade, staggered upgrade, IBM Dashboard, Call Home, Storage Insights, etc.

Typical first calls

1. Ask / infer version → list_versions() if unsure.

2. If the user named a command: find_docs_for_command(command="ceph fs volume create", version="ibm-9.1").

3. Else: search_docs(query="...", component="cephfs", version="ibm-9.1").

4. For snippets: search_examples(query="...", language="bash", component="cephadm").

5. get_doc_page / get_doc_chunk to read the full section.

Related MCP server: Markdown RAG MCP

Ceph Engineering Intelligence Platform

Setup

git clone https://github.com/pdhiran/ceph-document-kb.git
cd ceph-document-kb
pip install -e .

GitHub repository name is ceph-document-kb. The Python package and Cursor MCP key are ceph-doc-kb. Point cwd at this clone.

Indices under knowledge/ are committed so the MCP can serve immediately. Rebuild only when docs change (see Updating the knowledge base).

Incorporate into an agent

Cursor (stdio)

{
  "mcpServers": {
    "ceph-doc-kb": {
      "command": "python3",
      "args": ["-m", "ceph_doc_kb.server.mcp_server"],
      "cwd": "/path/to/ceph-document-kb"
    }
  }
}

The server discovers every knowledge/doc-*/ directory that has metadata.json (upstream + IBM) and merges search. Restart Cursor after editing mcp.json.

SSE

python3 -m ceph_doc_kb.server.mcp_server --transport sse --host 0.0.0.0 --port 8082

{
  "mcpServers": {
    "ceph-doc-kb": {
      "url": "http://localhost:8082/sse",
      "transport": "sse"
    }
  }
}

REST

python3 -m ceph_doc_kb.server.rest_api --host 0.0.0.0 --port 8100

curl "http://127.0.0.1:8100/api/search?query=erasure+coding&component=rados"

Endpoint reference and agent wrappers: BOB_INTEGRATION_GUIDE.md, examples/agent_integration.py. VS Code: vscode-extension/README.md.

Tool catalog

Component map

Results include source_file so you can tell IBM (ibm-docs/9.1/...) from upstream (rados/operations/pools.rst).

Agent workflow: IBM upgrade procedure

1. Confirm IBM version (8.1 vs 9.1) → version="ibm-9.1".

2. search_docs(query="staggered upgrade", component="cephadm", version="ibm-9.1")

3. get_doc_page / get_doc_chunk on the best hit.

4. Any CLI in the answer → ceph-cmd-kb verify_command(..., version="tentacle").

Agent workflow: command → docs → verify

1. ceph-cmd-kb verify_command (existence).

2. find_docs_for_command (procedure and caveats).

3. ceph-issue-kb search_issues if the user is debugging a failure of that command.

Updating the knowledge base

Same --since YYYY-MM-DD contract as python index_issues.py --since DATE.

Upstream RST

Needs a Ceph git checkout with doc/. Re-parses only RST files touched since the date and merges into the existing FAISS index.

# First-time full build
git clone --depth 1 --branch v20.2.1 --sparse https://github.com/ceph/ceph.git /tmp/ceph-docs
cd /tmp/ceph-docs && git sparse-checkout set doc
cd /path/to/ceph-document-kb
python3 index_docs.py --docs-path /tmp/ceph-docs/doc --version 20.2.1 --verbose

# Date delta (git log --since). Needs git history covering the date —
# a `--depth 1` clone from the full-build step above is not enough.
python3 index_docs.py --since 2026-08-01 \
    --docs-path /tmp/ceph-docs/doc \
    --repo-path /tmp/ceph-docs \
    --version 20.2.1 --verbose

# Tag-to-tag delta (git diff between tags)
python3 index_docs.py --update --docs-path /tmp/ceph-docs/doc \
    --repo-path /tmp/ceph-docs --from-version v20.2.1 --to-version v20.2.2

Requires an existing knowledge/doc-{version}/ for --since. Deleted RST files drop out of the index.

IBM HTML

IBM has no git history. --since recrawls the IBM docs API, hash-compares against --cache-dir, and skips the FAISS rebuild when HTML is unchanged. If pages changed, the IBM index is rebuilt from the current snapshot. Without --cache-dir, --since still recrawls and always full-rebuilds (no hash-diff).

python3 index_ibm_docs.py --version 9.1 --since 2026-08-01 \
    --cache-dir ./cache/ibm-9.1 --verbose

python3 index_ibm_docs.py --version 8.1 --verbose          # full
python3 index_ibm_docs.py --version 8.1 --max-pages 5      # smoke test

Supported IBM versions: 8.0, 8.1, 9.0, 9.1 (see IBM_VERSIONS in src/ceph_doc_kb/constants.py). Output: knowledge/doc-ibm-{version}/.

Maintainer wrapper

./update_index.sh                 # last run, or last 1 day
./update_index.sh 7
./update_index.sh 2026-08-01
./update_index.sh --reset

Environment: CEPH_DOCS_REPO (default /tmp/ceph-docs), CEPH_VERSION (default 20.2.1), IBM_VERSIONS (default 9.1), SKIP_UPSTREAM=1, SKIP_IBM=1.

Metadata fields: last_incremental_since on metadata.json; IBM also writes updated_since on ibm_crawl_metadata.json.

The MCP auto-pulls this git repo on a timer and hot-reloads every knowledge/doc-*/ index (Cursor stays open; only a .py pull respawns the MCP subprocess). ./update_index.sh touches .reload_trigger for the same in-process reload. A missing git remote skips pull but still watches the trigger. --no-auto-update disables both git pull and the trigger watcher (no pull, no trigger).

Full maintainer help: UPDATING.md.

Architecture

Upstream RST  → parser → scorer → embedder → knowledge/doc-20.2.1/<component>/
IBM HTML API  → crawler → ibm_parser ──────→ knowledge/doc-ibm-9.1/<component>/
                                              command_xref.json
                                                      │
                                                      ▼
                         search router (BM25 + FAISS + quality rank)

See DEVELOPMENT.md and SPEC.md.

Development

pip install -e ".[dev]"
pytest