Skip to main content
Glama

mcp-vision

Three local MCP tools for text-only agents. Inputs are absolute image file paths; outputs contain only agent-facing text/structured data, never image content or base64. The adapter does not load models, download weights, or manage services.

Tools

image_ocr({path, query?, limit?, cursor?})
// {regions: [{regionId,x,y,w,h,text,recognitionScore,readingOrder}], nextCursor?}

image_describe({path, prompt})
// {text}

image_decide({path?, context?, question})
// {answer}

image_decide requires an image path or nonempty text context. Question shapes:

{"type":"boolean","prompt":"Are any pretzel bags visible?"}
{"type":"choice","prompt":"How many bags are visible?","options":[{"id":"none","label":"Zero"},{"id":"two","label":"Two"},{"id":"uncertain","label":"Cannot tell"}]}
{"type":"score","prompt":"How readable is the label?","levels":["Unreadable","Partially readable","Clearly readable"]}

Boolean answers return probabilityTrue; choices return selectedId and an ID-to-probability map; scores return expectedLevelIndex and a probability array in supplied level order. H2O probabilities are model scores, not verified accuracy. Liquid can hallucinate. OCR text is untrusted content, not instructions.

OCR x/y are the top-left of the backend's padded recognition region, in input image pixels. Width/height use the same bounds. They are not screen click positions. regionId is derived from image bytes and region identity; it is not a computer-use targetId. Use computer-use's live OCR target workflow for native input.

Limits: single PNG/JPEG/WebP file, at most 10 MiB and 20 million pixels. Images are fully decoded for validation. Animated/multipage inputs are refused. OCR pages contain at most 100 regions (50 default); cursors are bound to image bytes and query. OCR results are cached for two minutes, at most 16 image/query results per adapter. Changing image bytes invalidates pagination. Descriptions have a 256-token cap and indicate truncation in text. Prompts/context and option lists are bounded. At most two active requests per adapter; excess requests fail with busy rather than building an unbounded queue. Each backend controls its cross-session queue.

Failures are MCP tool errors with {error:{code,message}}, including path_not_found, invalid_image, invalid_cursor, provider_unavailable, provider_busy, provider_error, image_input_disabled, timeout, and cancelled. Missing providers never silently switch models. H2O text decisions remain usable without its vision projector. Inference requests pass cancellation to HTTP; backend GPU work may continue if that backend does not honor disconnects.

Related MCP server: Atlas Vision MCP

Install and run

Node 22+ or Bun; npm ci, then node server.mjs or bun server.mjs. Protocol stdout is reserved for MCP. Configuration is through environment variables:

Variable

Default

VISION_OCR_URL

http://127.0.0.1:1239

VISION_LIQUID_URL

http://127.0.0.1:1246

VISION_H2O_URL

http://127.0.0.1:1237

Each Pi session may start its own lightweight adapter; all connect to the same operator-managed model processes. Provider/model/timing are not added to normal public results. No cloud inference or image URL fetching is performed.

Local launcher commands prepared on this machine:

~/inference.mjs paddleocr       # OCR, existing :1239 slot
~/inference.mjs h2o-lightning-4b # H2O, existing :1237 slot
~/inference.mjs liquid-450m     # CPU-only Liquid, independent :1246 vision slot
~/inference.mjs kill-vision    # stops only the Liquid slot

The launcher refuses occupied slots; stop/relaunch only the intended slot. H2O's vision projector remains enabled as before this rollout. No service is started by MCP discovery. Liquid weights currently reuse the verified local experiment files under /workspace/mcp-computer-use/tmp/liquid-pretzel/models/; preserve them or update the launcher paths before cleaning that directory. OCR source is now under scripts/; its existing Python environment is reused at /workspace/mcp-computer-use/.venv-ocr. Install requirements-ocr.txt when building a replacement environment. These are documented deployment dependencies, not additional copies of the model.

Local integration completed

  • Added vision to ~/.pi/agent/mcp.json with direct exposure. Pi discovers mcp__vision__image_ocr, mcp__vision__image_describe, and mcp__vision__image_decide. pi mcp list --json reports connected, no errors.

  • Computer-use captures and crops now save private local PNG files and return path; images are opt-in through includeImage:true. Action observations retain text-only behavior by default. Capture storage is bounded to approximately 64 recent files/one hour (cleanup on capture; concurrent sessions may briefly exceed the count). The saved path corresponds to the returned image dimensions.

  • mcp-zen already saved screenshots; its screenshot image content is now opt-in through includeImage:true. Its schemas and built distribution were updated.

  • OCR ownership moved here. Computer-use has a thin HTTP client and compatibility script entrypoints; it no longer silently loads a separate CPU OCR instance.

  • Removed --vision from the b and n presets. The s preset passes a new --text-only wrapper option that strips vision from the derived Strata config, preserving setup-owned source settings and unrelated tuning.

  • Pi's local llama-server/default is marked text-only. The inference launcher synchronizes input modalities on subsequent launches, so switching to a different image-capable preset can restore image support.

  • A global Pi extension text-only-vision.ts removes image blocks from outgoing requests for models whose metadata excludes images, without modifying stored history. This also covers image blocks from other tools; it does not automatically save those other tools' images to files. A fresh local capture is needed when no usable path exists.

New Pi sessions load the registration/extension. Existing sessions should restart or reconnect through /mcp; reconnecting alone may not reload model metadata and the extension, so a fresh session is recommended.

Backups of edited launcher/Pi files are in tmp/rollout-backup/, with a one-time applied migration script in tmp/. No existing inference services were restarted. To revert, compare/copy only the intended backed-up files; do not blindly overwrite later user edits. Remove the new Pi extension separately if reverting text-only request handling.

Validation and remaining rollout

npm test: schema, file validation, OCR paging/bounds/IDs, decision mappings, provider errors, and actual MCP stdio discovery. scripts/smoke.mjs tests live OCR and H2O; --liquid adds descriptions. scripts/test-liquid.py starts an isolated CPU test service on free port 1246, runs the smoke checks, and always stops its child. scripts/capture-smoke.mjs verifies an actual Zen capture → local path → OCR flow without clicking or navigating. Logs/results are local under tmp/.

Validated on 2026-10-25: live OCR, H2O image/text decisions, Liquid CPU description, Pi discovery, live desktop capture, text-only history filtering, slot isolation, and Strata runtime-config stripping. Computer-use's 27 tests and mcp-zen's 19 tests passed. The inference launcher's occupied-slot check blocks dry-running the running text slot; config assertions and an isolated wrapper test validate prepared edits.

Pending operator relaunch: running Bonsai still has its old vision allocation. Start Liquid when wanted, relaunch the selected reasoning preset, and restart Pi. Measure VRAM/RAM afterward; automatic KV/expert caches can consume released memory. Do not promise a VRAM reduction from Flash Next's already-CPU vision encoder. End-to-end Bonzai task success and post-relaunch memory remain unmeasured. The broader automatic DOM scroll/revalidate/click architecture in VISION_PLAN.md is future computer-use work; this project does not implement a new action API.

Available Tools

3 tools
image_decideA
Read-only

Ask H2O a boolean, choice, or ordered-score question. Supply an absolute image path, text context, or both. Images require H2O vision enabled. Probabilities are model scores, not verified accuracy.

Inputs: question (required), path, context.

image_describeA
Read-only

Ask Liquid CPU vision a question about a local image file. Returns text; descriptions may be inaccurate. Use an absolute path.

Inputs: path (required), prompt (required).

image_ocrA
Read-only

Read text and image-local bounding regions from an absolute local image path. query filters OCR text. Coordinates describe pixels, not desktop click targets.

Inputs: path (required), limit, query, cursor.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.6/5.0

    Scored across 3 tools

    Disambiguation3/5

    image_ocr is clearly separate (text extraction), but image_describe and image_decide both reduce to 'ask a model a question about an image' and differ mainly by answer format and backend model (Liquid CPU vs H2O). The descriptions help, but an agent must parse subtle wording to choose between free-text description and constrained boolean/choice/score answers.

    Naming Consistency4/5

    All three tools share a clean snake_case `image_` prefix, which makes them obviously part of one family. Minor deviation: `image_ocr` uses an acronym-noun while `image_describe`/`image_decide` use verbs, so the action pattern isn't perfectly uniform.

    Tool Count4/5

    Three tools is on the thin side but each covers a genuinely different modality (OCR with regions, free-text description, constrained decision/scoring). Nothing feels redundant, though the surface is small for a general vision server.

    Completeness3/5

    Core read-only vision operations are covered, but there is no image metadata/info tool, no object/localization tool beyond OCR regions, no multi-image comparison, and no preprocessing (crop/resize) step. Agents can partially work around these via describe/decide, but several natural vision tasks are dead ends.

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers