# mcp-local-vision

Lets a **text-only LLM read images and PDFs** by delegating to a vision model running on your own GPU, through `llama.cpp`.

Built for the case where your coding model has no vision at all — GLM on the Z.AI coding plan, DeepSeek, most local models. The coding model stays where it is; this server becomes its eyes.

- **No API keys, no cloud.** Images never leave the machine.
- **Free PDF text.** Pages with a text layer are extracted by poppler — zero model calls, exact output, instant.
- **Scanned pages fall back to vision.** Rendered at 200 DPI and read by the VLM automatically.
- **Gives the GPU back.** `llama-server` is started on demand and stopped after an idle period, so a shared card is not held hostage between screenshots.
- **Verbatim-first prompting.** Tuned to transcribe exactly, not to summarize helpfully.

## Why this exists

A model can only see an image if the *provider* serves a vision-capable model. On the Z.AI coding plan every model reports `attachment: false`:

```
zai-coding-plan:  glm-5.2, glm-5.2-highspeed, glm-4.7, glm-5-turbo   → text only
```

No prompt trick fixes that. Something with eyes has to do the looking, and on a 12GB card that something can be local.

## Requirements

| | |
|---|---|
| GPU | 8GB VRAM or more (developed on an RTX 3060 12GB) |
| `llama.cpp` | built with your GPU backend, providing `llama-server` |
| Model | a vision GGUF **plus its `mmproj` projector** |
| poppler | `sudo apt-get install -y poppler-utils` — needed only for scanned PDFs |
| Node | 20+ |

### Building llama.cpp with CUDA

llama.cpp ships no Linux CUDA prebuilt (the CUDA release assets are Windows-only), so build it:

```sh
sudo apt-get install -y cmake ninja-build gcc-13 g++-13 nvidia-cuda-toolkit
git clone --depth 1 https://github.com/ggml-org/llama.cpp ~/app/llama.cpp
cd ~/app/llama.cpp
cmake -S . -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=86 \          # 86 = RTX 30xx; 89 = 40xx; 120 = 50xx
  -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++-13 \
  -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_EXAMPLES=OFF
cmake --build build --target llama-server -j $(nproc)
```

`gcc-13` matters: Ubuntu's CUDA 12.4 `nvcc` rejects GCC 15, which is the system default on recent Ubuntu. Pointing `CMAKE_CUDA_HOST_COMPILER` at 13 is the whole fix.

### Getting the model

```sh
mkdir -p ~/models && cd ~/models
B=https://huggingface.co/unsloth/Qwen3-VL-4B-Instruct-GGUF/resolve/main
curl -L -O $B/Qwen3-VL-4B-Instruct-UD-Q4_K_XL.gguf         # 2.5GB — the language model
curl -L -o mmproj-4B-F16.gguf $B/mmproj-F16.gguf           # 0.8GB — the vision encoder
```

Both files are required. **Without `mmproj`, `llama-server` loads fine and is simply blind** — it will answer about an image it never saw. The projector stays at F16 on purpose: it is small, and quantizing the vision tower is what costs you fine print.

The 4B is the default because ~3.4GB of weights leaves the card usable for other work. Swap in the 8B (`unsloth/Qwen3-VL-8B-Instruct-GGUF`, 5.1GB + 1.2GB) via `VISION_MODEL_PATH` if the GPU is yours alone; the OCR gain is small (DocVQA 96.1 vs 95.3).

## Install

```sh
git clone https://github.com/jshsakura/mcp-local-vision ~/app/mcp-local-vision
cd ~/app/mcp-local-vision && npm install
npm test        # builds fixtures, then exercises every tool over real MCP
```

## Register with opencode

In `~/.config/opencode/opencode.json` (global) or `./opencode.json` (per project):

```json
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "local-vision": {
      "type": "local",
      "command": ["/home/you/app/mcp-local-vision/bin/launch.sh"],
      "enabled": true,
      "timeout": 15000
    }
  }
}
```

Use `bin/launch.sh` rather than `node src/index.mjs`: MCP clients spawn servers with a minimal environment, and if your node lives under nvm it will not be on `PATH`. The launcher finds it and survives nvm upgrades.

## Register with Claude Code

```sh
claude mcp add local-vision -s user -- /home/you/app/mcp-local-vision/bin/launch.sh
```

## Optional: a chat UI on the same model

Because `llama-server` speaks the OpenAI API, anything that talks to OpenAI can share it — including [Open WebUI](https://github.com/open-webui/open-webui), which gives you a browser chat window with image upload against the same local model your coding agent uses.

This only works with the **idle-sleep** setup below, not the managed-process fallback: a chat UI needs something listening even when nothing has been asked yet.

```sh
docker run -d --network=host \
  -e PORT=8090 \
  -e OPENAI_API_BASE_URL=http://127.0.0.1:8080/v1 \
  -e OPENAI_API_KEY=sk-local \
  -e ENABLE_OLLAMA_API=False \
  -v open-webui:/app/backend/data \
  --name open-webui --restart unless-stopped \
  ghcr.io/open-webui/open-webui:main
```

`--network=host` is required so the container can reach a `llama-server` bound to `127.0.0.1`; `PORT=8090` moves Open WebUI off its default 8080, which `llama-server` already holds. Open http://localhost:8090 and create the admin account on first visit.

The model appears as `qwen3-vl-4b-instruct` with `capabilities: ["completion","multimodal"]`, so the image attach button works. Idle polling from the UI does not wake the GPU.

### Serving the tools to Open WebUI as well

Open WebUI (v0.6.31+) speaks MCP, but **only over Streamable HTTP** — it cannot spawn a stdio server. Run this one in HTTP mode alongside the stdio one:

```sh
VISION_MANAGED=0 mcp-local-vision --http 38765
```

Then register it in **Admin Settings → External Tools → + Add Server**:

- **Connection Type: `MCP (Streamable HTTP)`** — this part matters. The field defaults to OpenAPI, and pointing an OpenAPI connection at an MCP server gives you a crash or an infinite loading spinner rather than a useful error.
- **URL:** `http://127.0.0.1:38765/mcp`

Only administrators can add MCP servers; the permission that lets ordinary users add OpenAPI tools does not extend to these.

To check the server rather than the UI, run the handshake by hand — a healthy server answers `200` with an SSE `event: message` frame:

```sh
curl -i -X POST http://127.0.0.1:38765/mcp \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"probe","version":"1"}}}'
```

**Know what this does and does not give you.** `read_image` takes a *path on the server's filesystem*. An image you drag into the chat window is an upload, not a path, so the tools cannot see it — that case is what the direct model connection above is for. The tools earn their keep on files already on disk ("read ~/docs/contract.pdf"), which is the same job they do in a coding agent. The two are complements, not alternatives.

**Pass absolute paths over HTTP.** In stdio mode the client spawns the server, so a relative path resolves against the client's working directory. Over HTTP the server is a long-lived process with a working directory of its own — under systemd, your home — and a relative path quietly resolves there instead. The tool descriptions say so, but it is worth knowing when a file "is not found" that plainly exists.

The HTTP endpoint has no authentication of its own and these tools read the filesystem, so it binds to `127.0.0.1` by default. If you move it off loopback with `VISION_HTTP_HOST`, set `VISION_ALLOWED_ROOTS` too.

## Tools

### `read_image(path, question?)`

Reads png, jpg, webp, gif, bmp, tiff, avif, heic. EXIF rotation is applied, transparency is flattened onto white (dark-mode screenshots with alpha otherwise read as black-on-black), and the long edge is capped at 1568px.

Omit `question` for a full transcription. Pass one when you only need a single detail — the model writes a short answer instead of a whole document, which keeps both latency and returned context small.

### `read_pdf(path, pages?, question?, mode?)`

- `pages` — `"1-5"`, `"2,7,9"`, `"3"`. Omit for the whole document.
- `mode` — `auto` (default: text layer where present, vision for scanned pages), `text` (never call the model), `vision` (render every page — use when the text layer is garbled or you need to read figures).

Text-layer pages are **not capped**; a 200-page report comes back in about a second. Only pages that need the vision model are capped (`VISION_MAX_PAGES`, default 5), and any page dropped by the cap is named in the output so the model knows to ask again with `pages` narrowed.

### `vision_doctor()`

Reports the binary/model/projector paths and whether each exists, server state, and whether poppler is installed. Call it first when something fails.

## GPU lifecycle

There are two ways to keep the GPU free between uses. **Prefer the first.**

### llama.cpp's own idle sleep (recommended)

Recent `llama-server` takes `--sleep-idle-seconds N`: after N seconds with no task it unloads the model and KV cache, and the next request reloads it. The process stays up, so the endpoint answers the whole time — which is what lets a chat UI sit connected without pinning the card.

`GET /health`, `/props` and `/models` are explicitly exempt: they neither wake it nor reset the idle timer. A UI polling for the model list therefore costs nothing.

Measured on a 3060 12GB with the 4B model:

| State | llama-server VRAM |
|---|---|
| Loaded | 4676 MiB |
| Asleep | **126 MiB** (CUDA context only) |

Waking costs ~5.7s on the next call. Run it under systemd and point this server at it with `VISION_MANAGED=0`:

```ini
# ~/.config/systemd/user/llama-vision.service
[Service]
Type=exec
ExecStart=/home/you/app/llama.cpp/build/bin/llama-server \
  --model /home/you/models/Qwen3-VL-4B-Instruct-UD-Q4_K_XL.gguf \
  --mmproj /home/you/models/mmproj-4B-F16.gguf \
  --alias qwen3-vl-4b-instruct \
  --host 127.0.0.1 --port 8080 --ctx-size 8192 \
  --sleep-idle-seconds 300 --no-warmup
Restart=on-failure

[Install]
WantedBy=default.target
```

```sh
systemctl --user daemon-reload && systemctl --user enable --now llama-vision
sudo loginctl enable-linger $USER    # survive logout
```

### Process lifecycle managed here (fallback)

If your `llama-server` predates `--sleep-idle-seconds`, leave `VISION_MANAGED=1` (the default). This process then starts `llama-server` on the first vision call and kills it after `VISION_IDLE_MS`. It frees VRAM just as well, but the endpoint is *gone* while idle — fine for MCP, useless for a chat UI that expects something listening.

Either way the idle timer resets on every call, so a burst of reads never expires mid-flow.

Two things keep the managed mode from crashing on a busy card:

- **`-ngl` is not passed by default.** Forcing `-ngl 99` disables llama.cpp's automatic layer fitting — it logs `n_gpu_layers already set by user to 99, abort` and then dies with a CUDA OOM rather than offloading the overflow to CPU. Left unset, it sizes itself to whatever VRAM is actually free.
- **Startup failures explain themselves.** llama-server's stderr is captured, and an OOM is reported as an OOM with the tail of its log, not as `exited with code 1`.

If you would rather run `llama-server` yourself under systemd or in a tmux pane, set `VISION_MANAGED=0` and this process will only ever health-check it.

## Configuration

All optional, all via environment variables.

| Variable | Default | Purpose |
|---|---|---|
| `VISION_SERVER_URL` | `http://127.0.0.1:8080` | llama-server base URL |
| `VISION_MANAGED` | `1` | Start/stop llama-server here. `0` = you run it |
| `VISION_IDLE_MS` | `300000` | Release the GPU after this idle time. `0` = never |
| `VISION_SERVER_BIN` | `~/app/llama.cpp/build/bin/llama-server` | Binary to launch |
| `VISION_MODEL_PATH` | `~/models/Qwen3-VL-4B-Instruct-UD-Q4_K_XL.gguf` | Language model GGUF |
| `VISION_MMPROJ_PATH` | `~/models/mmproj-4B-F16.gguf` | Vision projector GGUF |
| `VISION_GPU_LAYERS` | unset | `-ngl`. **Leave unset** — setting it disables auto-fit |
| `VISION_CONTEXT` | `8192` | `-c` context size |
| `VISION_EXTRA_ARGS` | — | Extra llama-server flags, space separated |
| `VISION_MAX_EDGE` | `1568` | Long-edge cap in px. Raise for dense small print |
| `VISION_MAX_TOKENS` | `4096` | Output cap per call |
| `VISION_PDF_DPI` | `200` | Rasterization DPI for scanned pages |
| `VISION_MAX_PAGES` | `5` | Cap on *vision* pages per call |
| `VISION_TIMEOUT_MS` | `300000` | Per-request timeout |
| `VISION_STARTUP_TIMEOUT_MS` | `180000` | How long to wait for a cold load |
| `VISION_MAX_FILE_BYTES` | `209715200` | Reject larger inputs |
| `VISION_TEXT_PAGE_MIN_CHARS` | `50` | Below this, a PDF page counts as scanned |
| `VISION_ALLOWED_ROOTS` | unset | Colon-separated directory allowlist. Unset = unrestricted |
| `VISION_VERBOSE` | `0` | Pass llama-server stderr through |
| `VISION_API_KEY` | unset | Bearer token, if llama-server runs with `--api-key` |
| `VISION_HTTP_PORT` | unset | Serve MCP over Streamable HTTP on this port instead of stdio |
| `VISION_HTTP_HOST` | `127.0.0.1` | Bind address for HTTP mode |
| `VISION_HTTP_TOKEN` | unset | Bearer token required on `/mcp`. Mandatory off loopback |
| `VISION_NODE` | unset | Explicit node path for `bin/launch.sh` |

## Model choice

Any vision GGUF with a matching `mmproj` works. Qwen3-VL is the default because its OCR is the strongest of the small models (DocVQA 96.1 at 8B) and it handles CJK well.

| Model | Size | Notes |
|---|---|---|
| Qwen3-VL-4B-Instruct UD-Q4_K_XL | 2.5GB + 0.8GB mmproj | Default. DocVQA 95.3 |
| Qwen3-VL-8B-Instruct UD-Q4_K_XL | 5.1GB + 1.2GB mmproj | DocVQA 96.1, if the card is free |
| MiniCPM-V 4.5 | ~6GB | Similar profile |

Use the **Instruct** build, not the Thinking one. Thinking builds spend thousands of tokens reasoning before they transcribe — far slower for what is fundamentally a transcription job.

## Limits

- One image per call; multi-image comparison is not exposed.
- Video is not supported even though the model accepts it.
- Handwriting is materially worse than print.
- The server reads any file the user running it can read unless `VISION_ALLOWED_ROOTS` is set.

## License

MIT