# knowledge-index (`ki`)

A private, self-hosted **RAG service exposed over MCP** (Model Context Protocol), built as a
replacement for NotebookLM in a personal AI-tooling ecosystem: documents stay on hardware I
control, retrieval is hybrid (BM25 + dense vectors fused with RRF), and every capability is
reachable as a tool from a Claude Code session instead of a web UI.

> **Project status — read this first.** The ingestion, chunking, retrieval and embedding layers are
> implemented and covered by **130 passing unit tests** under `ruff` + `mypy --strict`. The storage
> layer (PostgreSQL schema/migrations and the OpenSearch index + RRF pipeline) is written and its
> integration tests exist, but its **live acceptance criteria are still unverified**: the host that
> runs OpenSearch and Postgres has been offline, so the end-to-end pipeline has not been exercised
> against real services. This is **not a deployed product** — it is a working codebase with a
> deliberate architecture, and I would rather say so than imply otherwise.

---

## Why it exists

Hosted notebook tools are convenient but impose three costs I did not want to pay: documents leave
my machine, the tool cannot be driven programmatically, and scraping around those limits violates
the terms of service. `ki` trades that convenience for control — the corpus lives on my own host,
and the interface is MCP, so an agent can ingest, query and manage notebooks as ordinary tool calls.

## What it does

- **Ingests** documents (PDF, Markdown, TXT today; URL, YouTube, Google Docs, DOCX planned) into
  isolated *notebooks*.
- **Retrieves** with a hybrid strategy: BM25 lexical search and dense k-NN vector search, fused by
  **Reciprocal Rank Fusion**, then optionally re-ranked with a cross-encoder.
- **Exposes 14 MCP tools** over SSE, consumed directly from agent sessions.
- **Abstracts the embedding provider** behind one interface — Cohere (default), Voyage, OpenAI,
  Gemini — so the model is a configuration decision, not an architectural one.

---

## Architecture

```
SOURCE            PDF · Markdown · TXT
   |
   v
ADAPTERS          pypdf primary, pdfplumber fallback when extraction yields <100 chars/page
                  de-hyphenation · whitespace normalization · encoding cascade (utf-8-sig → cp1252 → latin-1)
                  page_map preserved (page, char_start, char_end) for citations
   |
   v
CHUNKING          structure-aware splitting · content_hash for deduplication
   |
   v
EMBEDDINGS        provider registry behind one ABC · 1024-dim vectors
                  notebook freezes its provider+model at creation time
   |
   v
STORAGE           PostgreSQL  → notebooks, sources, chunks, jobs, costs (5 tables, 8 indices,
                                 NOTIFY + updated_at triggers, Alembic migrations)
                  OpenSearch  → ki_chunks index, HNSW (lucene, cosine, 1024) + BM25
   |
   v
RETRIEVAL         hybrid query → RRF fusion → optional cross-encoder rerank
   |
   v
INTERFACE         14 MCP tools over SSE
```

**Stack:** Python 3.11 · Pydantic · asyncio · OpenSearch · PostgreSQL + Alembic · Cohere ·
`pypdf`/`pdfplumber` · pytest · ruff · mypy (strict).

---

## Design decisions and trade-offs

The interesting part of this project is not the code, it is what got ruled out and why.

### Vector store: reuse OpenSearch instead of adding Qdrant or pgvector

**Chosen:** the OpenSearch cluster already running on my host, with a dedicated index.

- *Qdrant* has better ergonomics and native payload filtering, but it means standing up another
  service at roughly 500 MB of RAM on hardware that is already tight.
- *pgvector* adds zero infrastructure, but HNSW performance degrades past ~100k chunks, lexical
  search via `pg_trgm` is materially weaker than BM25, and there is no native RRF.

OpenSearch gives BM25 and k-NN in one engine with RRF available in the search pipeline, at no
additional memory cost. The price paid: the index mapping fixes vector dimensionality at **1024**,
which constrains which embedding models are usable, and the cluster is shared — so isolation is by
index name with `number_of_shards=1`.

### Embeddings: multi-provider abstraction, Cohere as default

Self-hosting `bge-m3` was the theoretically better answer (no runtime cost, nothing leaves the host)
and was **measured and rejected**: the available hardware has 8 GB of RAM and no usable GPU, and the
model's working memory would starve the OpenSearch instance sharing the box. Gemini's embeddings are
768-dimensional and would not fit the index without a full rebuild.

So the provider sits behind an ABC with a registry, and each notebook **freezes its provider and
model at creation time** — mixing embedding spaces inside one index silently destroys retrieval
quality, and freezing makes that failure impossible rather than merely discouraged. Migration is
handled explicitly by a re-index operation.

### Query returns chunks, not a synthesized answer

`ki_query` returns ranked chunks; synthesis is opt-in through a separate tool. The consumer is
already an LLM session, so synthesizing server-side would mean paying for a second model call to
produce something the caller can do for free — and it would force notebooks marked *sensitive*
through a cloud provider they are specifically configured to avoid.

### Privacy as an enforced constraint, not a convention

Notebooks can be flagged **sensitive**. The provider layer rejects any provider whose `is_cloud`
flag is true for those notebooks — enforced in the abstraction, with parametric tests covering it,
rather than left to the caller to remember.

---

## Measured numbers

| Metric | Value |
|---|---|
| Unit tests | **130**, passing |
| Type checking | `mypy --strict`, clean |
| Lint | `ruff`, clean |
| Embedding latency (Cohere `embed-multilingual-v3.0`) | **~1150 ms p95** (target was <2 s) |
| Embedding dimensionality | 1024, confirmed against the live API |
| Embedding providers implemented | 4 (1 fully live, 3 behind the same interface) |
| PostgreSQL schema | 5 tables · 8 indices · 2 triggers |

The rerank path is deliberately disabled by default in development: the Cohere trial allows
**10 rerank calls per month** against 1000 embedding calls, which is trivially exhausted during
active retrieval work. Finding that in the response headers before it caused a mid-development
outage is exactly the kind of cost detail that separates a demo from something operable.

---

## Running it

```bash
pip install -e ".[dev]"
cp .env.example .env.local        # fill in COHERE_API_KEY and KI_PG_DSN

# Unit tests — no network, no external services required
pytest tests/unit -v

# Live integration tests — require reachable OpenSearch + PostgreSQL and real credentials
KI_TEST_LIVE=1 pytest tests/integration -v
```

Bootstrapping the storage layer (both idempotent, safe to re-run):

```bash
alembic upgrade head              # PostgreSQL: schema, tables, indices, triggers
python scripts/init_opensearch.py # OpenSearch: ki_chunks index + RRF search pipeline
```

### Configuration

| Variable | Purpose |
|---|---|
| `COHERE_API_KEY` | Embedding provider credential |
| `KI_PG_DSN` | libpq DSN; rewritten internally to `+asyncpg` / `+psycopg` |
| `KI_PG_SCHEMA` | Defaults to `ki` |
| `KI_OPENSEARCH_URL` | e.g. `http://localhost:9200` |
| `KI_EMBEDDING_PROVIDER` | `cohere` (default), `voyage`, `openai`, `gemini` |
| `KI_TEST_LIVE` | Set to `1` to run integration tests against real services |

Secrets are typed as Pydantic `SecretStr` so they are never emitted through logs or reprs.

---

## What I would do next

- Verify the live acceptance criteria for the storage layer once the host is back, and run the
  end-to-end smoke over a real corpus.
- Add retrieval quality evaluation — a ground-truth query set with recall@k, so changes to chunking
  or fusion can be judged by measurement instead of impression.
- Add tracing over the ingestion and query paths; per-notebook cost accounting is already modelled
  in the schema but not yet surfaced.

## License

Not currently licensed for reuse. Published as a portfolio artifact.