```markdown
# Notes RAG on PostgreSQL + pgvector + MCP

Migrates the retrieval layer of [AskMyNotes](https://github.com/00aimlds00/class-notes-rag) from flat files (NumPy + pickle) to a real PostgreSQL database with the [pgvector](https://github.com/pgvector/pgvector) extension, running in Docker. The database is exposed to Claude through a custom [MCP](https://modelcontextprotocol.io) (Model Context Protocol) server, so questions can be answered via genuine SQL-backed semantic search instead of an in-memory NumPy comparison.

## Architecture

```
Existing notes index (embeddings.npy, chunks.pkl, metadata.json)
        │
        ▼
   migrate.py  ──────────────►  PostgreSQL + pgvector (Docker)
                                        │
                                        ▼
                              mcp_server.py (MCP tools)
                                        │
                                        ▼
                         mcp-client  ──────────►  Claude
```

## Prerequisites

- **Docker Desktop** — running and verified working (`docker run hello-world`)
- **[uv](https://docs.astral.sh/uv/)** — Python package/environment manager
- **An Anthropic API key** — [console.anthropic.com](https://console.anthropic.com)
- **A pre-built notes index from [class-notes-rag](https://github.com/00aimlds00/class-notes-rag)** — this project migrates that data, it doesn't generate it. Clone that repo, add your own notes to `notes/`, and run its `ingest.py` first to produce `index/embeddings.npy`, `index/chunks.pkl`, and `index/metadata.json`.
- **[mcp-client](https://github.com/00aimlds00/mcp-client)** — used to actually connect to and query the MCP server built here.

This repo, `class-notes-rag`, and `mcp-client` are designed to sit as **sibling folders** on disk, e.g.:
```
C:\Users\you\class-notes-rag\
C:\Users\you\notes-rag-pg\        ← this repo
C:\Users\you\mcp-client\
```

## Setup

1. Clone this repo and start the database:
   ```
   git clone https://github.com/00aimlds00/notes-rag-pg.git
   cd notes-rag-pg
   docker compose up -d
   ```
   This runs PostgreSQL 16 with pgvector pre-installed, on host port **5433** (not 5432 — chosen to avoid clashing with any existing native PostgreSQL install on your machine; change it in `docker-compose.yml` if 5433 is also taken).

2. Install Python dependencies:
   ```
   uv sync
   ```

3. Create a `.env` file in this folder:
   ```
   DATABASE_URL=postgresql://notesuser:notespass@localhost:5433/notesdb
   ANTHROPIC_API_KEY=your-api-key-here
   ```

4. Migrate your notes index (requires `class-notes-rag/index/` to already exist, as a sibling directory — see Prerequisites):
   ```
   uv run migrate.py
   ```
   This creates a `notes_chunks` table and loads every chunk and embedding. Safe to re-run — it drops and recreates the table each time.

## Usage

**Test retrieval directly**, without MCP or Claude, to confirm the database is working:
```
uv run test_search.py
```

**Connect via MCP**, from your `mcp-client` project:
```
cd ../mcp-client
uv add psycopg2-binary sentence-transformers
uv run client.py ../notes-rag-pg/mcp_server.py
```
(The `uv add` step is needed because `mcp-client` launches this server as a subprocess using its own Python environment — see Troubleshooting below.)

Then ask a question, e.g. `What is a patent?` — Claude will call the `search_notes` tool, which runs a live cosine-distance SQL query against Postgres.

## Tools exposed by the MCP server

| Tool | Description |
|---|---|
| `search_notes(query, top_k=4, class_filter="")` | Semantic search over indexed notes. `class_filter` can restrict to `"class-xi"` or `"class-xii"`. |
| `list_note_sources(class_filter="")` | Lists all distinct indexed source files. |

## Design notes

- **No approximate-nearest-neighbor index (ivfflat/hnsw) is used.** At this data scale (a few thousand chunks), exact brute-force cosine search is both fast enough and fully accurate. An `ivfflat` index was tested and found to *reduce* retrieval accuracy at this size — see full write-up in the project documentation.
- **Uses the public `pgvector/pgvector:pg16` image unmodified** — no custom Docker image to build or pull separately.
- **`mcp_server.py` disables Hugging Face/Transformers stdout output** (`HF_HUB_DISABLE_PROGRESS_BARS`, etc.) and lazy-loads `sentence-transformers`, since MCP's stdio transport uses stdout exclusively for JSON-RPC — any stray print statement would corrupt the protocol stream.

## Troubleshooting

- **`password authentication failed`** — likely a port conflict with an existing native PostgreSQL install. Change the host-side port in `docker-compose.yml` (`"5433:5432"` → e.g. `"5434:5432"`) and update `DATABASE_URL` to match.
- **MCP client shows `Connection closed` / handshake timeout** — the server process almost certainly crashed on import. Make sure `mcp-client`'s own environment has `psycopg2-binary` and `sentence-transformers` installed (see Usage above) — it launches the server using its own Python interpreter, not this project's.
- **Search returns irrelevant results** — if you've added a vector index, try dropping it (`DROP INDEX ...;`) and re-testing with `test_search.py`. Approximate indexes need tuning (`probes`) that isn't worth it below tens of thousands of rows.

## License

MIT
```