# Notebook Library MCP Server

Token-efficient document retrieval for substrate AI agents. Drop PDFs, text files, and markdown into notebook folders — they get chunked, embedded, and indexed for semantic search. Queries return only the most relevant passages (~2,500 tokens) instead of loading entire documents (50,000+).

## What It Does

Your AI agent gets a `notebook_library` tool with these actions:

| Action | Description |
|--------|-------------|
| `list_notebooks` | See all available notebooks |
| `create_notebook` | Create a new notebook collection |
| `query_notebook` | Semantic search within a notebook (the main one!) |
| `browse_notebook` | List documents in a notebook |
| `read_document` | Deep-read a specific document chunk by chunk |
| `notebook_stats` | Get statistics about a notebook |
| `sync_notebook` | Re-sync after adding/changing files |
| `remove_document` | Remove a document from the search index |

**Supported file formats:** `.pdf`, `.txt`, `.md`, `.text`, `.markdown`

## Architecture

```
data/
├── notebooks/               # Your document folders
│   ├── Research_Papers/     # Each subfolder = one notebook
│   │   ├── paper1.pdf
│   │   └── notes.md
│   └── Business_Docs/
│       └── plan.txt
└── notebook_chromadb/       # Vector database (auto-created)
    └── manifests/           # File change tracking

mcp_servers/
└── notebook_library/
    ├── server.py              # MCP server (if running standalone)
    ├── notebook_manager.py    # Core: ChromaDB ingestion + search
    ├── document_processor.py  # Text extraction + chunking
    ├── file_watcher.py        # Auto-ingestion on file changes
    └── requirements.txt

backend/tools/
├── notebook_library_tool.py        # Tool wrapper for consciousness loop
└── notebook_library_tool_schema.json  # Tool schema definition
```

**Embedding strategy (multi-tier fallback):**
1. **Hugging Face** (`jinaai/jina-embeddings-v2-base-de`) — local, free, multilingual
2. **Ollama** (`nomic-embed-text`) — local fallback if HF fails

No external API keys needed. Everything runs locally.

## Setup Guide

### 1. Install Dependencies

From your substrate root:

```bash
pip install -r mcp_servers/notebook_library/requirements.txt
```

Key dependencies:
- `chromadb==0.4.18` — vector database
- `transformers` + `torch` — Hugging Face embeddings (primary)
- `ollama` — embedding fallback
- `PyMuPDF` — PDF text extraction
- `watchdog` — file system monitoring

> **Note:** First run will download the Hugging Face embedding model (~270MB). This is a one-time download.

### 2. Create Data Directories

```bash
mkdir -p data/notebooks
mkdir -p data/notebook_chromadb
```

### 3. Copy the MCP Server Files

Copy the entire `mcp_servers/notebook_library/` directory into your substrate:

```
your_substrate/
└── mcp_servers/
    └── notebook_library/
        ├── __init__.py
        ├── server.py
        ├── notebook_manager.py
        ├── document_processor.py
        ├── file_watcher.py
        └── requirements.txt
```

### 4. Copy the Tool Wrapper

Copy these two files into your `backend/tools/` directory:

**`backend/tools/notebook_library_tool.py`** — The tool function your consciousness loop calls. This imports `NotebookManager` directly (no subprocess).

**`backend/tools/notebook_library_tool_schema.json`** — The tool schema so your agent knows how to call it.

### 5. Register the Tool in Your Consciousness Loop

Three integration points:

#### a) Import in `integration_tools.py`

Add to your imports:

```python
from tools.notebook_library_tool import notebook_library_tool as _notebook_library_tool
```

Add the wrapper method to your `IntegrationTools` class:

```python
def notebook_library(self, **kwargs) -> Dict[str, Any]:
    """
    Notebook Library — token-efficient document retrieval.
    """
    try:
        result = _notebook_library_tool(**kwargs)
        return result
    except Exception as e:
        return {
            "status": "error",
            "message": f"Notebook library error: {str(e)}"
        }
```

Add `'notebook_library_tool'` to your tool schema loading list so the JSON schema gets picked up.

#### b) Add tool call handler in `consciousness_loop.py`

In your tool execution block (where you handle `elif tool_name == "..."` cases), add:

```python
elif tool_name == "notebook_library":
    result = self.tools.notebook_library(**arguments)
```

#### c) Verify schema loading

The tool schema file (`notebook_library_tool_schema.json`) must be in `backend/tools/` alongside your other tool schemas. The schema loader should pick it up automatically if it follows the same pattern as your other tools.

### 6. Add Documents

Create notebook folders and drop files in:

```bash
mkdir -p data/notebooks/My_Research
cp ~/some_paper.pdf data/notebooks/My_Research/
cp ~/notes.md data/notebooks/My_Research/
```

Documents are auto-ingested when your agent first queries the notebook, or you can trigger a manual sync via the `sync_notebook` action.

## Environment Variables (Optional)

All have sensible defaults. Override only if needed:

| Variable | Default | Description |
|----------|---------|-------------|
| `NOTEBOOK_LIBRARY_PATH` | `data/notebooks` | Where notebook folders live |
| `NOTEBOOK_CHROMADB_PATH` | `data/notebook_chromadb` | Vector database storage |
| `OLLAMA_BASE_URL` | `http://192.168.2.175:11434` | Ollama server (fallback embeddings) |
| `OLLAMA_EMBEDDING_MODEL` | `nomic-embed-text` | Ollama model name |
| `NOTEBOOK_CHUNK_SIZE` | `2000` | Characters per chunk |
| `NOTEBOOK_CHUNK_OVERLAP` | `200` | Overlap between chunks |

**Important:** Update `OLLAMA_BASE_URL` to point to your own Ollama instance if you're using the Ollama fallback. The default points to the original developer's local network.

## How It Works

1. **Ingestion:** Documents are split into chunks (~2000 chars each with 200 char overlap), embedded using Hugging Face or Ollama, and stored in ChromaDB collections (one per notebook).

2. **Querying:** Your agent's query gets embedded with the same model, then ChromaDB finds the most similar chunks via cosine similarity. Only the top N passages are returned (default 5).

3. **File tracking:** A manifest system (MD5 hashes) tracks which files have been ingested. Changed files get re-processed; unchanged files are skipped.

4. **File watching:** A watchdog-based file watcher monitors notebook folders and auto-ingests new/modified files with a 2-second debounce.

## Example Agent Usage

Once integrated, your agent can use it like:

```
# List what's available
notebook_library(action="list_notebooks")

# Search for something specific
notebook_library(action="query_notebook", notebook="Research_Papers", query="transformer attention mechanisms")

# Browse a notebook's contents
notebook_library(action="browse_notebook", notebook="Research_Papers")

# Deep-read a specific document
notebook_library(action="read_document", notebook="Research_Papers", filename="paper1.pdf")

# Create a new notebook
notebook_library(action="create_notebook", name="Meeting_Notes", description="Weekly team meetings")
```

## Troubleshooting

**"No notebooks found"** — Make sure `data/notebooks/` exists and has at least one subfolder with files in it.

**Slow first query** — The first query to a notebook triggers ingestion (chunking + embedding all documents). Subsequent queries are fast. For large collections, run `sync_notebook` first.

**Embedding model download** — First run downloads the Jina embeddings model (~270MB). If this fails behind a firewall, the system falls back to Ollama. Make sure either HF model access or an Ollama instance is available.

**ChromaDB version mismatch** — Pin to `chromadb==0.4.18`. Newer versions may have breaking API changes.

**OLLAMA_BASE_URL** — If you see Ollama connection errors and you're not using Ollama, that's fine — it's just the fallback failing after HF already succeeded. If HF also fails, update this URL to your Ollama instance.