# PageIndex MCP (self-hosted)

A self-hosted MCP server exposing [PageIndex](https://github.com/VectifyAI/PageIndex)'s
vectorless, reasoning-based document retrieval. The `pageindex/` directory is a
vendored copy of VectifyAI's open-source PageIndex package (MIT licensed, see
`pageindex/LICENSE.upstream`).

How it works:

- **Ingest** (`app/ingest.py`): builds a hierarchical "table of contents" tree
  for a PDF using an LLM (Gemini Flash by default, configurable via
  `PAGEINDEX_MODEL` / `pageindex/config.yaml` + LiteLLM). This costs a small
  amount of LLM usage, once per document.
- **Hybrid OCR** (`app/ocr.py`): pages whose embedded text layer is too sparse
  (scans, image-heavy slides) are rendered and transcribed by a vision model
  during ingest; born-digital text pages are read losslessly for free. The
  transcriptions are cached so retrieval serves them too. Disable with
  `PAGEINDEX_OCR_MODEL=off`.
- **Figures** (`app/media.py`): after the tree is built, every page is
  rendered to a JPEG and a vision model (`PAGEINDEX_FIGURE_MODEL`, default
  gemini/gemini-3.8-flash, one call per page) marks its figures - diagrams,
  plots, sketches, photos - with a bounding box and a caption. Each figure is
  cropped from the vector PDF at 200 DPI. Logos, icons and plain text are
  skipped. Documents ingested before this existed get their figures via the
  web UI ("Extract figures") or `POST /api/media/backfill`; their trees are
  left untouched. Disable with `PAGEINDEX_FIGURE_MODEL=off`.
- **Serve** (`app/server.py`): exposes `list_documents`, `get_document`,
  `get_document_structure`, `get_page_content`, `list_figures`,
  `get_document_image` and `get_page_image` as MCP tools over streamable
  HTTP, protected by a bearer token. `get_page_content` lists the figures of
  each returned page (id, kind, caption). The connecting agent (e.g. Claude) does
  the navigation/reasoning itself - serving is free after ingest.
- **Text files**: anything that isn't a PDF (code, Jupyter notebooks,
  markdown, any UTF-8 file up to 10 MB) is stored as plain text without LLM
  ingest - instantly available, zero cost. The same MCP tools serve them,
  with 1-indexed line numbers taking the role of page numbers (e.g.
  `get_page_content(doc_id, "1-200")` returns the first 200 lines).
  Notebook outputs are stripped on upload; only markdown and code cells are
  kept.
- **Web UI** (`/`): minimal document manager - create folders (projects),
  upload PDFs and text files (PDF ingest runs in background workers), watch
  queued/processing status, rename and delete documents. Clicking a document
  opens a detail view (description, dates) with a button to open the original
  PDF/file in a new tab. Unlock with the same bearer token; it is kept in the
  browser's localStorage.

## Setup

1. Copy `.env.example` to `.env` and fill in:
   - `GEMINI_API_KEY` - used only during ingest (tree building + OCR)
   - `PAGEINDEX_MCP_API_KEY` - bearer token clients must send, e.g. `openssl rand -hex 32`
   - optional: `PAGEINDEX_MODEL` (any LiteLLM model id for tree-building;
     default gemini/gemini-3.5-flash), `PAGEINDEX_OCR_MODEL` (vision model
     for text-poor pages, "off" to disable), `PAGEINDEX_INGEST_WORKERS`
     (parallel PDF ingests, default 2) and `PAGEINDEX_MAX_CONCURRENT_LLM`
     (global cap on simultaneous LLM calls, default 8 - protects against
     provider rate limits; if an ingest still hits a rate limit or the
     model is overloaded, it is re-queued automatically with a growing
     cooldown)

2. Build and start:

   ```bash
   docker compose up -d --build
   ```

3. Upload PDFs and text files via the web UI at `https://<your-domain>/`
   (unlock with the `PAGEINDEX_MCP_API_KEY`). PDF ingest runs in the
   background; the list shows processing/done/failed per document. Text
   files are done immediately.

   Alternatively via CLI inside the container:

   ```bash
   docker compose exec pageindex-mcp python3 app/ingest.py /data/pdfs/lecture01.pdf --project "Machine Learning"
   ```

   Trees are saved to `<data>/trees/<doc_id>.json` and registered in
   `<data>/documents.json`.

## Figure API

All endpoints need the bearer token.

- `GET /api/documents/<doc_id>/figures` - `{"figures": [{id, page, kind, caption}]}`
- `GET /api/documents/<doc_id>/figures/<id>/image` - figure crop (JPEG), e.g. id `p12-1`
- `GET /api/documents/<doc_id>/pages/<n>/image` - rendered page (JPEG, rendered on first request)
- `POST /api/documents/<doc_id>/media` - (re-)extract the figures of one document
- `POST /api/media/backfill` - extract figures for every finished PDF that has none yet;
  body `{"project": "Deep Learning"}` restricts it to a folder, `{"force": true}` redoes all

Files live under `<data>/media/<doc_id>/`.

## Connecting an MCP client

```json
{
  "mcpServers": {
    "pageindex-self": {
      "type": "http",
      "url": "https://<your-domain>/mcp",
      "headers": {
        "Authorization": "Bearer <PAGEINDEX_MCP_API_KEY>"
      }
    }
  }
}
```

For Claude Code:

```bash
claude mcp add --transport http pageindex-self https://<your-domain>/mcp \
  --header "Authorization: Bearer <PAGEINDEX_MCP_API_KEY>"
```

For opencode (`~/.config/opencode/opencode.json`, or a project-level
`opencode.json`):

```json
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "pageindex-self": {
      "type": "remote",
      "url": "https://<your-domain>/mcp",
      "enabled": true,
      "headers": {
        "Authorization": "Bearer <PAGEINDEX_MCP_API_KEY>"
      }
    }
  }
}
```

To keep the token out of the config file, opencode supports env substitution:
`"Authorization": "Bearer {env:PAGEINDEX_MCP_API_KEY}"`.

## Deployment

The compose file attaches the service to the external `dokploy-network`, so in
Dokploy you only need to add a domain pointing at service `pageindex-mcp`,
port `8000` (Traefik handles TLS). The container port is intentionally not
published on the host - the bearer token must only travel over HTTPS.

For plain local use (no Dokploy), swap the `networks` section for the
commented-out `127.0.0.1` port binding in `docker-compose.yml`.

`GET /health` is unauthenticated and returns `ok` - useful for uptime checks.

## Persistence

`../files/data/` (PDFs, generated trees, registry) is bind-mounted and persists
across rebuilds/restarts. On Dokploy this is the app's `files` storage dir,
which survives redeploys (the code dir does not). Back it up if you don't want
to re-run ingest.