knowledge-index (ki)

A private, self-hosted RAG service exposed over MCP (Model Context Protocol), built as a replacement for NotebookLM in a personal AI-tooling ecosystem: documents stay on hardware I control, retrieval is hybrid (BM25 + dense vectors fused with RRF), and every capability is reachable as a tool from a Claude Code session instead of a web UI.

Project status — read this first. The ingestion, chunking, retrieval and embedding layers are implemented and covered by 130 passing unit tests under ruff + mypy --strict. The storage layer (PostgreSQL schema/migrations and the OpenSearch index + RRF pipeline) is written and its integration tests exist, but its live acceptance criteria are still unverified: the host that runs OpenSearch and Postgres has been offline, so the end-to-end pipeline has not been exercised against real services. This is not a deployed product — it is a working codebase with a deliberate architecture, and I would rather say so than imply otherwise.

Why it exists

Hosted notebook tools are convenient but impose three costs I did not want to pay: documents leave my machine, the tool cannot be driven programmatically, and scraping around those limits violates the terms of service. ki trades that convenience for control — the corpus lives on my own host, and the interface is MCP, so an agent can ingest, query and manage notebooks as ordinary tool calls.

What it does

• Ingests documents (PDF, Markdown, TXT today; URL, YouTube, Google Docs, DOCX planned) into isolated notebooks.

• Retrieves with a hybrid strategy: BM25 lexical search and dense k-NN vector search, fused by Reciprocal Rank Fusion, then optionally re-ranked with a cross-encoder.

• Exposes 14 MCP tools over SSE, consumed directly from agent sessions.

• Abstracts the embedding provider behind one interface — Cohere (default), Voyage, OpenAI, Gemini — so the model is a configuration decision, not an architectural one.

Architecture

SOURCE            PDF · Markdown · TXT
   |
   v
ADAPTERS          pypdf primary, pdfplumber fallback when extraction yields <100 chars/page
                  de-hyphenation · whitespace normalization · encoding cascade (utf-8-sig → cp1252 → latin-1)
                  page_map preserved (page, char_start, char_end) for citations
   |
   v
CHUNKING          structure-aware splitting · content_hash for deduplication
   |
   v
EMBEDDINGS        provider registry behind one ABC · 1024-dim vectors
                  notebook freezes its provider+model at creation time
   |
   v
STORAGE           PostgreSQL  → notebooks, sources, chunks, jobs, costs (5 tables, 8 indices,
                                 NOTIFY + updated_at triggers, Alembic migrations)
                  OpenSearch  → ki_chunks index, HNSW (lucene, cosine, 1024) + BM25
   |
   v
RETRIEVAL         hybrid query → RRF fusion → optional cross-encoder rerank
   |
   v
INTERFACE         14 MCP tools over SSE

Stack: Python 3.11 · Pydantic · asyncio · OpenSearch · PostgreSQL + Alembic · Cohere · pypdf/pdfplumber · pytest · ruff · mypy (strict).

Design decisions and trade-offs

The interesting part of this project is not the code, it is what got ruled out and why.

Vector store: reuse OpenSearch instead of adding Qdrant or pgvector

Chosen: the OpenSearch cluster already running on my host, with a dedicated index.

• Qdrant has better ergonomics and native payload filtering, but it means standing up another service at roughly 500 MB of RAM on hardware that is already tight.

• pgvector adds zero infrastructure, but HNSW performance degrades past ~100k chunks, lexical search via pg_trgm is materially weaker than BM25, and there is no native RRF.

OpenSearch gives BM25 and k-NN in one engine with RRF available in the search pipeline, at no additional memory cost. The price paid: the index mapping fixes vector dimensionality at 1024, which constrains which embedding models are usable, and the cluster is shared — so isolation is by index name with number_of_shards=1.

Embeddings: multi-provider abstraction, Cohere as default

Self-hosting bge-m3 was the theoretically better answer (no runtime cost, nothing leaves the host) and was measured and rejected: the available hardware has 8 GB of RAM and no usable GPU, and the model's working memory would starve the OpenSearch instance sharing the box. Gemini's embeddings are 768-dimensional and would not fit the index without a full rebuild.

So the provider sits behind an ABC with a registry, and each notebook freezes its provider and model at creation time — mixing embedding spaces inside one index silently destroys retrieval quality, and freezing makes that failure impossible rather than merely discouraged. Migration is handled explicitly by a re-index operation.

Query returns chunks, not a synthesized answer

ki_query returns ranked chunks; synthesis is opt-in through a separate tool. The consumer is already an LLM session, so synthesizing server-side would mean paying for a second model call to produce something the caller can do for free — and it would force notebooks marked sensitive through a cloud provider they are specifically configured to avoid.

Privacy as an enforced constraint, not a convention

Notebooks can be flagged sensitive. The provider layer rejects any provider whose is_cloud flag is true for those notebooks — enforced in the abstraction, with parametric tests covering it, rather than left to the caller to remember.

Measured numbers

The rerank path is deliberately disabled by default in development: the Cohere trial allows 10 rerank calls per month against 1000 embedding calls, which is trivially exhausted during active retrieval work. Finding that in the response headers before it caused a mid-development outage is exactly the kind of cost detail that separates a demo from something operable.

Running it

pip install -e ".[dev]"
cp .env.example .env.local        # fill in COHERE_API_KEY and KI_PG_DSN

# Unit tests — no network, no external services required
pytest tests/unit -v

# Live integration tests — require reachable OpenSearch + PostgreSQL and real credentials
KI_TEST_LIVE=1 pytest tests/integration -v

Bootstrapping the storage layer (both idempotent, safe to re-run):

alembic upgrade head              # PostgreSQL: schema, tables, indices, triggers
python scripts/init_opensearch.py # OpenSearch: ki_chunks index + RRF search pipeline

Configuration

Secrets are typed as Pydantic SecretStr so they are never emitted through logs or reprs.

What I would do next

• Verify the live acceptance criteria for the storage layer once the host is back, and run the end-to-end smoke over a real corpus.

• Add retrieval quality evaluation — a ground-truth query set with recall@k, so changes to chunking or fusion can be judged by measurement instead of impression.

• Add tracing over the ingestion and query paths; per-notebook cost accounting is already modelled in the schema but not yet surfaced.

License

Not currently licensed for reuse. Published as a portfolio artifact.