Skip to main content
Glama
jshsakura

mcp-local-vision

by jshsakura
OverviewSchemaSourceRelated ServersDiscussionsAdmin
JavaScript
Local

mcp-local-vision

Lets a text-only LLM read images and PDFs by delegating to a vision model running on your own GPU, through llama.cpp.

Built for the case where your coding model has no vision at all — GLM on the Z.AI coding plan, DeepSeek, most local models. The coding model stays where it is; this server becomes its eyes.

  • No API keys, no cloud. Images never leave the machine.

  • Free PDF text. Pages with a text layer are extracted by poppler — zero model calls, exact output, instant.

  • Scanned pages fall back to vision. Rendered at 200 DPI and read by the VLM automatically.

  • Gives the GPU back. llama-server is started on demand and stopped after an idle period, so a shared card is not held hostage between screenshots.

  • Verbatim-first prompting. Tuned to transcribe exactly, not to summarize helpfully.

Why this exists

A model can only see an image if the provider serves a vision-capable model. On the Z.AI coding plan every model reports attachment: false:

zai-coding-plan:  glm-5.2, glm-5.2-highspeed, glm-4.7, glm-5-turbo   → text only

No prompt trick fixes that. Something with eyes has to do the looking, and on a 12GB card that something can be local.

Related MCP server: image-recognition-mcp

Requirements

GPU

8GB VRAM or more (developed on an RTX 3060 12GB)

llama.cpp

built with your GPU backend, providing llama-server

Model

a vision GGUF plus its mmproj projector

poppler

sudo apt-get install -y poppler-utils — needed only for scanned PDFs

Node

20+

Building llama.cpp with CUDA

llama.cpp ships no Linux CUDA prebuilt (the CUDA release assets are Windows-only), so build it:

sudo apt-get install -y cmake ninja-build gcc-13 g++-13 nvidia-cuda-toolkit
git clone --depth 1 https://github.com/ggml-org/llama.cpp ~/app/llama.cpp
cd ~/app/llama.cpp
cmake -S . -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=86 \          # 86 = RTX 30xx; 89 = 40xx; 120 = 50xx
  -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++-13 \
  -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_EXAMPLES=OFF
cmake --build build --target llama-server -j $(nproc)

gcc-13 matters: Ubuntu's CUDA 12.4 nvcc rejects GCC 15, which is the system default on recent Ubuntu. Pointing CMAKE_CUDA_HOST_COMPILER at 13 is the whole fix.

Getting the model

mkdir -p ~/models && cd ~/models
B=https://huggingface.co/unsloth/Qwen3-VL-4B-Instruct-GGUF/resolve/main
curl -L -O $B/Qwen3-VL-4B-Instruct-UD-Q4_K_XL.gguf         # 2.5GB — the language model
curl -L -o mmproj-4B-F16.gguf $B/mmproj-F16.gguf           # 0.8GB — the vision encoder

Both files are required. Without mmproj, llama-server loads fine and is simply blind — it will answer about an image it never saw. The projector stays at F16 on purpose: it is small, and quantizing the vision tower is what costs you fine print.

The 4B is the default because ~3.4GB of weights leaves the card usable for other work. Swap in the 8B (unsloth/Qwen3-VL-8B-Instruct-GGUF, 5.1GB + 1.2GB) via VISION_MODEL_PATH if the GPU is yours alone; the OCR gain is small (DocVQA 96.1 vs 95.3).

Install

git clone https://github.com/jshsakura/mcp-local-vision ~/app/mcp-local-vision
cd ~/app/mcp-local-vision && npm install
npm test        # builds fixtures, then exercises every tool over real MCP

Register with opencode

In ~/.config/opencode/opencode.json (global) or ./opencode.json (per project):

{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "local-vision": {
      "type": "local",
      "command": ["/home/you/app/mcp-local-vision/bin/launch.sh"],
      "enabled": true,
      "timeout": 15000
    }
  }
}

Use bin/launch.sh rather than node src/index.mjs: MCP clients spawn servers with a minimal environment, and if your node lives under nvm it will not be on PATH. The launcher finds it and survives nvm upgrades.

Register with Claude Code

claude mcp add local-vision -s user -- /home/you/app/mcp-local-vision/bin/launch.sh

Optional: a chat UI on the same model

Because llama-server speaks the OpenAI API, anything that talks to OpenAI can share it — including Open WebUI, which gives you a browser chat window with image upload against the same local model your coding agent uses.

This only works with the idle-sleep setup below, not the managed-process fallback: a chat UI needs something listening even when nothing has been asked yet.

docker run -d --network=host \
  -e PORT=8090 \
  -e OPENAI_API_BASE_URL=http://127.0.0.1:8080/v1 \
  -e OPENAI_API_KEY=sk-local \
  -e ENABLE_OLLAMA_API=False \
  -v open-webui:/app/backend/data \
  --name open-webui --restart unless-stopped \
  ghcr.io/open-webui/open-webui:main

--network=host is required so the container can reach a llama-server bound to 127.0.0.1; PORT=8090 moves Open WebUI off its default 8080, which llama-server already holds. Open http://localhost:8090 and create the admin account on first visit.

The model appears as qwen3-vl-4b-instruct with capabilities: ["completion","multimodal"], so the image attach button works. Idle polling from the UI does not wake the GPU.

Serving the tools to Open WebUI as well

Open WebUI (v0.6.31+) speaks MCP, but only over Streamable HTTP — it cannot spawn a stdio server. Run this one in HTTP mode alongside the stdio one:

VISION_MANAGED=0 mcp-local-vision --http 38765

Then register it in Admin Settings → External Tools → + Add Server:

  • Connection Type: MCP (Streamable HTTP) — this part matters. The field defaults to OpenAPI, and pointing an OpenAPI connection at an MCP server gives you a crash or an infinite loading spinner rather than a useful error.

  • URL: http://127.0.0.1:38765/mcp

Only administrators can add MCP servers; the permission that lets ordinary users add OpenAPI tools does not extend to these.

To check the server rather than the UI, run the handshake by hand — a healthy server answers 200 with an SSE event: message frame:

curl -i -X POST http://127.0.0.1:38765/mcp \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"probe","version":"1"}}}'

Know what this does and does not give you. read_image takes a path on the server's filesystem. An image you drag into the chat window is an upload, not a path, so the tools cannot see it — that case is what the direct model connection above is for. The tools earn their keep on files already on disk ("read ~/docs/contract.pdf"), which is the same job they do in a coding agent. The two are complements, not alternatives.

Pass absolute paths over HTTP. In stdio mode the client spawns the server, so a relative path resolves against the client's working directory. Over HTTP the server is a long-lived process with a working directory of its own — under systemd, your home — and a relative path quietly resolves there instead. The tool descriptions say so, but it is worth knowing when a file "is not found" that plainly exists.

The HTTP endpoint has no authentication of its own and these tools read the filesystem, so it binds to 127.0.0.1 by default. If you move it off loopback with VISION_HTTP_HOST, set VISION_ALLOWED_ROOTS too.

Tools

read_image(path, question?)

Reads png, jpg, webp, gif, bmp, tiff, avif, heic. EXIF rotation is applied, transparency is flattened onto white (dark-mode screenshots with alpha otherwise read as black-on-black), and the long edge is capped at 1568px.

Omit question for a full transcription. Pass one when you only need a single detail — the model writes a short answer instead of a whole document, which keeps both latency and returned context small.

read_pdf(path, pages?, question?, mode?)

  • pages — "1-5", "2,7,9", "3". Omit for the whole document.

  • mode — auto (default: text layer where present, vision for scanned pages), text (never call the model), vision (render every page — use when the text layer is garbled or you need to read figures).

Text-layer pages are not capped; a 200-page report comes back in about a second. Only pages that need the vision model are capped (VISION_MAX_PAGES, default 5), and any page dropped by the cap is named in the output so the model knows to ask again with pages narrowed.

vision_doctor()

Reports the binary/model/projector paths and whether each exists, server state, and whether poppler is installed. Call it first when something fails.

GPU lifecycle

There are two ways to keep the GPU free between uses. Prefer the first.

llama.cpp's own idle sleep (recommended)

Recent llama-server takes --sleep-idle-seconds N: after N seconds with no task it unloads the model and KV cache, and the next request reloads it. The process stays up, so the endpoint answers the whole time — which is what lets a chat UI sit connected without pinning the card.

GET /health, /props and /models are explicitly exempt: they neither wake it nor reset the idle timer. A UI polling for the model list therefore costs nothing.

Measured on a 3060 12GB with the 4B model:

State

llama-server VRAM

Loaded

4676 MiB

Asleep

126 MiB (CUDA context only)

Waking costs ~5.7s on the next call. Run it under systemd and point this server at it with VISION_MANAGED=0:

# ~/.config/systemd/user/llama-vision.service
[Service]
Type=exec
ExecStart=/home/you/app/llama.cpp/build/bin/llama-server \
  --model /home/you/models/Qwen3-VL-4B-Instruct-UD-Q4_K_XL.gguf \
  --mmproj /home/you/models/mmproj-4B-F16.gguf \
  --alias qwen3-vl-4b-instruct \
  --host 127.0.0.1 --port 8080 --ctx-size 8192 \
  --sleep-idle-seconds 300 --no-warmup
Restart=on-failure

[Install]
WantedBy=default.target
systemctl --user daemon-reload && systemctl --user enable --now llama-vision
sudo loginctl enable-linger $USER    # survive logout

Process lifecycle managed here (fallback)

If your llama-server predates --sleep-idle-seconds, leave VISION_MANAGED=1 (the default). This process then starts llama-server on the first vision call and kills it after VISION_IDLE_MS. It frees VRAM just as well, but the endpoint is gone while idle — fine for MCP, useless for a chat UI that expects something listening.

Either way the idle timer resets on every call, so a burst of reads never expires mid-flow.

Two things keep the managed mode from crashing on a busy card:

  • -ngl is not passed by default. Forcing -ngl 99 disables llama.cpp's automatic layer fitting — it logs n_gpu_layers already set by user to 99, abort and then dies with a CUDA OOM rather than offloading the overflow to CPU. Left unset, it sizes itself to whatever VRAM is actually free.

  • Startup failures explain themselves. llama-server's stderr is captured, and an OOM is reported as an OOM with the tail of its log, not as exited with code 1.

If you would rather run llama-server yourself under systemd or in a tmux pane, set VISION_MANAGED=0 and this process will only ever health-check it.

Configuration

All optional, all via environment variables.

Variable

Default

Purpose

VISION_SERVER_URL

http://127.0.0.1:8080

llama-server base URL

VISION_MANAGED

1

Start/stop llama-server here. 0 = you run it

VISION_IDLE_MS

300000

Release the GPU after this idle time. 0 = never

VISION_SERVER_BIN

~/app/llama.cpp/build/bin/llama-server

Binary to launch

VISION_MODEL_PATH

~/models/Qwen3-VL-4B-Instruct-UD-Q4_K_XL.gguf

Language model GGUF

VISION_MMPROJ_PATH

~/models/mmproj-4B-F16.gguf

Vision projector GGUF

VISION_GPU_LAYERS

unset

-ngl. Leave unset — setting it disables auto-fit

VISION_CONTEXT

8192

-c context size

VISION_EXTRA_ARGS

—

Extra llama-server flags, space separated

VISION_MAX_EDGE

1568

Long-edge cap in px. Raise for dense small print

VISION_MAX_TOKENS

4096

Output cap per call

VISION_PDF_DPI

200

Rasterization DPI for scanned pages

VISION_MAX_PAGES

5

Cap on vision pages per call

VISION_TIMEOUT_MS

300000

Per-request timeout

VISION_STARTUP_TIMEOUT_MS

180000

How long to wait for a cold load

VISION_MAX_FILE_BYTES

209715200

Reject larger inputs

VISION_TEXT_PAGE_MIN_CHARS

50

Below this, a PDF page counts as scanned

VISION_ALLOWED_ROOTS

unset

Colon-separated directory allowlist. Unset = unrestricted

VISION_VERBOSE

0

Pass llama-server stderr through

VISION_API_KEY

unset

Bearer token, if llama-server runs with --api-key

VISION_HTTP_PORT

unset

Serve MCP over Streamable HTTP on this port instead of stdio

VISION_HTTP_HOST

127.0.0.1

Bind address for HTTP mode

VISION_HTTP_TOKEN

unset

Bearer token required on /mcp. Mandatory off loopback

VISION_NODE

unset

Explicit node path for bin/launch.sh

Model choice

Any vision GGUF with a matching mmproj works. Qwen3-VL is the default because its OCR is the strongest of the small models (DocVQA 96.1 at 8B) and it handles CJK well.

Model

Size

Notes

Qwen3-VL-4B-Instruct UD-Q4_K_XL

2.5GB + 0.8GB mmproj

Default. DocVQA 95.3

Qwen3-VL-8B-Instruct UD-Q4_K_XL

5.1GB + 1.2GB mmproj

DocVQA 96.1, if the card is free

MiniCPM-V 4.5

~6GB

Similar profile

Use the Instruct build, not the Thinking one. Thinking builds spend thousands of tokens reasoning before they transcribe — far slower for what is fundamentally a transcription job.

Limits

  • One image per call; multi-image comparison is not exposed.

  • Video is not supported even though the model accepts it.

  • Handwriting is materially worse than print.

  • The server reads any file the user running it can read unless VISION_ALLOWED_ROOTS is set.

License

MIT

Available Tools

3 tools
read_imageRead an imageA
Read-only

Read a local image file (png, jpg, webp, gif, bmp, tiff, avif, heic) with a local vision model. Returns a verbatim transcription of all visible text plus a structural description of UI, charts, and diagrams. Use this for screenshots, error dialogs, design mockups, whiteboard photos, and graphs. Supply question to ask about one detail instead of transcribing the whole image.

ParametersJSON Schema
NameRequiredDescriptionDefault
pathYesAbsolute or relative path to the image file on this machine.
questionNoOptional specific question about the image. Omit for a full description.

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the readOnlyHint annotation, the description adds meaningful behavioral details: processing happens through a local vision model, output is a verbatim transcription plus structural description, and supported image formats are specified. It does not contradict the annotations, and the local-model detail adds useful context about privacy and execution.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three sentences deliver purpose, output, supported formats, use cases, and question semantics without fluff. The most important information is front-loaded and every sentence contributes.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool is moderately complex (vision model, multiple formats, optional question), but the description covers return values, use cases, and parameter behavior. Since there is no output schema, the description properly explains what the agent can expect. Combined with strong annotations, no critical gaps remain.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3. The description clarifies that supplying `question` returns a focused detail instead of full transcription, but the schema already states 'Optional specific question about the image. Omit for a full description,' making this marginal added value.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb ('Read') and resource ('local image file'), enumerates supported formats, and states the concrete output: verbatim text transcription plus structural description. This clearly distinguishes it from sibling tools like read_pdf, which handles PDFs.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicit use cases are listed ('screenshots, error dialogs, design mockups, whiteboard photos, and graphs') and parameter usage for focused questions is described. It provides clear context for when to use the tool but does not explicitly mention when not to use it or name alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

read_pdfRead a PDFA
Read-only

Read a local PDF. Pages with a text layer are extracted exactly and for free; scanned or image-only pages are rendered and read by a local vision model. Use pages (e.g. "1-5" or "2,7") to limit the work on long documents, and question to search for one fact instead of transcribing everything.

ParametersJSON Schema
NameRequiredDescriptionDefault
modeNoauto (default): text layer where present, vision for scanned pages. text: never call the model — fastest, exact, but blind to figures. vision: render every page and read it visually — use when the text layer is garbled or you need to understand figures.
pathYesAbsolute or relative path to the PDF file on this machine.
pagesNo1-based page selector, e.g. "1-5", "2,7,9", or "3". Omit for all pages (capped).
questionNoOptional specific question. Omit for a full transcription.

TDQS

A4.4/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description goes well beyond the readOnlyHint annotation by disclosing the hybrid mechanism: text-layer pages are extracted exactly, while scanned pages are rendered and read by a local vision model. It also mentions the cost implication ('for free') and the optional question-based search, providing valuable behavioral context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences, front-loaded with the purpose, and each sentence earns its place. It is efficient and avoids redundancy with the schema.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity and absence of an output schema, the description covers the main behavior and usage well. It implies the output (text transcription or answer) but does not explicitly describe the return structure. Overall, it is sufficiently complete for most use cases.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The description adds usage examples for pages and question, but the schema already provides similar details, so the added value is marginal. It does not significantly enhance parameter understanding beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly identifies the tool as reading a local PDF, with a specific verb and resource. It distinguishes from sibling tools by focusing on PDFs, and explains the dual text/vision behavior, which sets it apart from read_image and vision_doctor.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context for using the tool (local PDFs, page selection, question-based extraction) but does not explicitly mention exclusions or alternative tools. The guidance on when to use pages and question is helpful, but it lacks an explicit 'when not to use' or alternative tool comparison.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

vision_doctorCheck the local vision setupA
Read-only

Report whether Ollama is reachable, whether the configured vision model is pulled, and whether poppler is installed for PDF rendering. Call this first when read_image or read_pdf fails.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

A4.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description adds concrete behavioral details—exactly what it checks (Ollama, vision model, poppler)—beyond the readOnlyHint annotation. It does not mention potential side effects, but the annotation already establishes it as read-only. It could be more explicit about output format, but the described reporting behavior is clear.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences: first states the functional output, second gives usage guidance. Every sentence is informative and there is no redundant information. It is front-loaded and easy to understand.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool has no parameters, no output schema, and simple diagnostics, the description provides sufficient context: what it reports, when to call it, and its relationship to siblings. It is complete enough for an agent to decide to use it and understand its high-level behavior.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

There are zero parameters, so the baseline for parameter semantics is 4. The description does not need to explain parameters, and it adds value by explaining the tool's purpose and usage, which indirectly clarifies the empty schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses the specific verb 'Report' and clearly defines what is reported: Ollama reachability, vision model pulled status, and poppler installation. This distinguishes it from sibling tools read_image and read_pdf by making it a diagnostic tool rather than a content reader.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly states when to use: 'Call this first when read_image or read_pdf fails.' This provides a clear trigger and implies that read_image/read_pdf are the primary tools, with this tool serving as a troubleshooting step. No alternative is needed beyond the siblings already listed.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 3 tool updatesv0.1.0
    • First observedread_image
    • First observedread_pdf
    • First observedvision_doctor
Deploy Server
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

A4.4/5.0

Scored across 3 tools

Disambiguation5/5

Each tool targets a distinct input type: images, PDFs, and system diagnostics. The descriptions make boundaries clear, with no overlapping purposes.

Naming Consistency4/5

Two tools follow the read_* pattern, but 'vision_doctor' deviates from the verb_noun style. Still, it's readable and predictable.

Tool Count5/5

Three tools is well-scoped for a local vision server: two primary operations and one diagnostic helper. No redundancy or bloat.

Completeness4/5

Image and PDF reading are covered, along with setup diagnostics. Minor gaps like a standalone model list tool exist, but the core workflow is complete.

Maintenance

ActivitySlowing
ResponsivenessNo issues

Resources

Related MCP Connectors

Related MCP Servers