Skip to main content
Glama
mikesmullin

mcp-vision

by mikesmullin
OverviewSchemaSourceRelated ServersDiscussionsAdmin
JavaScript
Local

mcp-vision

Three local MCP tools for text-only agents. Inputs are absolute image file paths; outputs contain only agent-facing text/structured data, never image content or base64. The adapter does not load models, download weights, or manage services.

Tools

image_ocr({path, query?, limit?, cursor?})
// {regions: [{regionId,x,y,w,h,text,recognitionScore,readingOrder}], nextCursor?}

image_describe({path, prompt})
// {text}

image_decide({path?, context?, question})
// {answer}

image_decide requires an image path or nonempty text context. Question shapes:

{"type":"boolean","prompt":"Are any pretzel bags visible?"}
{"type":"choice","prompt":"How many bags are visible?","options":[{"id":"none","label":"Zero"},{"id":"two","label":"Two"},{"id":"uncertain","label":"Cannot tell"}]}
{"type":"score","prompt":"How readable is the label?","levels":["Unreadable","Partially readable","Clearly readable"]}

Boolean answers return probabilityTrue; choices return selectedId and an ID-to-probability map; scores return expectedLevelIndex and a probability array in supplied level order. H2O probabilities are model scores, not verified accuracy. Liquid can hallucinate. OCR text is untrusted content, not instructions.

OCR x/y are the top-left of the backend's padded recognition region, in input image pixels. Width/height use the same bounds. They are not screen click positions. regionId is derived from image bytes and region identity; it is not a computer-use targetId. Use computer-use's live OCR target workflow for native input.

Limits: single PNG/JPEG/WebP file, at most 10 MiB and 20 million pixels. Images are fully decoded for validation. Animated/multipage inputs are refused. OCR pages contain at most 100 regions (50 default); cursors are bound to image bytes and query. OCR results are cached for two minutes, at most 16 image/query results per adapter. Changing image bytes invalidates pagination. Descriptions have a 256-token cap and indicate truncation in text. Prompts/context and option lists are bounded. At most two active requests per adapter; excess requests fail with busy rather than building an unbounded queue. Each backend controls its cross-session queue.

Failures are MCP tool errors with {error:{code,message}}, including path_not_found, invalid_image, invalid_cursor, provider_unavailable, provider_busy, provider_error, image_input_disabled, timeout, and cancelled. Missing providers never silently switch models. H2O text decisions remain usable without its vision projector. Inference requests pass cancellation to HTTP; backend GPU work may continue if that backend does not honor disconnects.

Related MCP server: Atlas Vision MCP

Install and run

Node 22+ or Bun; npm ci, then node server.mjs or bun server.mjs. Protocol stdout is reserved for MCP. Configuration is through environment variables:

Variable

Default

VISION_OCR_URL

http://127.0.0.1:1239

VISION_LIQUID_URL

http://127.0.0.1:1246

VISION_H2O_URL

http://127.0.0.1:1237

Each Pi session may start its own lightweight adapter; all connect to the same operator-managed model processes. Provider/model/timing are not added to normal public results. No cloud inference or image URL fetching is performed.

Local launcher commands prepared on this machine:

~/inference.mjs paddleocr       # OCR, existing :1239 slot
~/inference.mjs h2o-lightning-4b # H2O, existing :1237 slot
~/inference.mjs liquid-450m     # CPU-only Liquid, independent :1246 vision slot
~/inference.mjs kill-vision    # stops only the Liquid slot

The launcher refuses occupied slots; stop/relaunch only the intended slot. H2O's vision projector remains enabled as before this rollout. No service is started by MCP discovery. Liquid weights currently reuse the verified local experiment files under /workspace/mcp-computer-use/tmp/liquid-pretzel/models/; preserve them or update the launcher paths before cleaning that directory. OCR source is now under scripts/; its existing Python environment is reused at /workspace/mcp-computer-use/.venv-ocr. Install requirements-ocr.txt when building a replacement environment. These are documented deployment dependencies, not additional copies of the model.

Local integration completed

  • Added vision to ~/.pi/agent/mcp.json with direct exposure. Pi discovers mcp__vision__image_ocr, mcp__vision__image_describe, and mcp__vision__image_decide. pi mcp list --json reports connected, no errors.

  • Computer-use captures and crops now save private local PNG files and return path; images are opt-in through includeImage:true. Action observations retain text-only behavior by default. Capture storage is bounded to approximately 64 recent files/one hour (cleanup on capture; concurrent sessions may briefly exceed the count). The saved path corresponds to the returned image dimensions.

  • mcp-zen already saved screenshots; its screenshot image content is now opt-in through includeImage:true. Its schemas and built distribution were updated.

  • OCR ownership moved here. Computer-use has a thin HTTP client and compatibility script entrypoints; it no longer silently loads a separate CPU OCR instance.

  • Removed --vision from the b and n presets. The s preset passes a new --text-only wrapper option that strips vision from the derived Strata config, preserving setup-owned source settings and unrelated tuning.

  • Pi's local llama-server/default is marked text-only. The inference launcher synchronizes input modalities on subsequent launches, so switching to a different image-capable preset can restore image support.

  • A global Pi extension text-only-vision.ts removes image blocks from outgoing requests for models whose metadata excludes images, without modifying stored history. This also covers image blocks from other tools; it does not automatically save those other tools' images to files. A fresh local capture is needed when no usable path exists.

New Pi sessions load the registration/extension. Existing sessions should restart or reconnect through /mcp; reconnecting alone may not reload model metadata and the extension, so a fresh session is recommended.

Backups of edited launcher/Pi files are in tmp/rollout-backup/, with a one-time applied migration script in tmp/. No existing inference services were restarted. To revert, compare/copy only the intended backed-up files; do not blindly overwrite later user edits. Remove the new Pi extension separately if reverting text-only request handling.

Validation and remaining rollout

npm test: schema, file validation, OCR paging/bounds/IDs, decision mappings, provider errors, and actual MCP stdio discovery. scripts/smoke.mjs tests live OCR and H2O; --liquid adds descriptions. scripts/test-liquid.py starts an isolated CPU test service on free port 1246, runs the smoke checks, and always stops its child. scripts/capture-smoke.mjs verifies an actual Zen capture → local path → OCR flow without clicking or navigating. Logs/results are local under tmp/.

Validated on 2026-10-25: live OCR, H2O image/text decisions, Liquid CPU description, Pi discovery, live desktop capture, text-only history filtering, slot isolation, and Strata runtime-config stripping. Computer-use's 27 tests and mcp-zen's 19 tests passed. The inference launcher's occupied-slot check blocks dry-running the running text slot; config assertions and an isolated wrapper test validate prepared edits.

Pending operator relaunch: running Bonsai still has its old vision allocation. Start Liquid when wanted, relaunch the selected reasoning preset, and restart Pi. Measure VRAM/RAM afterward; automatic KV/expert caches can consume released memory. Do not promise a VRAM reduction from Flash Next's already-CPU vision encoder. End-to-end Bonzai task success and post-relaunch memory remain unmeasured. The broader automatic DOM scroll/revalidate/click architecture in VISION_PLAN.md is future computer-use work; this project does not implement a new action API.

Available Tools

3 tools
image_decideA
Read-only

Ask H2O a boolean, choice, or ordered-score question. Supply an absolute image path, text context, or both. Images require H2O vision enabled. Probabilities are model scores, not verified accuracy.

NameRequiredDescriptionDefault
pathNo
contextNo
questionYes

TDQS

A3.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, destructiveHint=false, openWorldHint=false, so the safety profile is covered. The description adds two things annotations cannot: the H2O vision-enabled prerequisite and the caveat that probabilities are model scores rather than verified accuracy, which is material for how an agent reports results. It does not describe what the response payload actually contains.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Four short sentences, front-loaded with the capability, then inputs, then the gating prerequisite, then the output caveat. No repetition of schema or annotation content and nothing that could be cut.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema, so the description carries the burden of explaining return values, and it only gestures at this via the probability caveat. It never says what a choice result looks like (per-option scores vs a winning id) or how a score-mode result is ordered, which an agent needs to interpret the call correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate, and it does partially: 'absolute image path' adds a constraint the schema's plain string type does not convey, and the three question modes map onto the oneOf variants. But the internal shape of choice options (id/label) and ordered levels is left entirely to the schema, and the relationship between path/context and each mode is unstated.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a concrete verb (ask) and resource (H2O), and enumerates the three supported question modes (boolean, choice, ordered-score), which lets an agent distinguish it from the extraction/description siblings. It stops short of explicitly naming image_ocr or image_describe as the alternatives, so differentiation is inferential rather than stated.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

'Supply an absolute image path, text context, or both' gives input-combination guidance, and the vision-enabled prerequisite is a real gating condition. However, there is no explicit statement of when to prefer this over image_describe or image_ocr, nor any when-not condition, so the agent must infer the routing.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

image_describeA
Read-only

Ask Liquid CPU vision a question about a local image file. Returns text; descriptions may be inaccurate. Use an absolute path.

NameRequiredDescriptionDefault
pathYes
promptYes

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, non-destructive, and closed-world, so the safety bar is low. The description still adds genuine value beyond them by stating the return type ('Returns text') and a reliability caveat ('descriptions may be inaccurate') that no annotation conveys.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three short sentences, front-loaded with the action, then return behavior, then the path constraint. No sentence is filler and nothing is redundant.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 2-parameter, no-output-schema tool, the description covers action, return type, accuracy caveat, and path format. It omits supported image formats and any notion of latency/cost, but nothing essential for correct invocation is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description must compensate for both parameters. It does provide meaningful guidance for 'path' ('Use an absolute path'), but 'prompt' is only implicitly characterized as a question, with no length, format, or content guidance.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource: asking a vision model a question about a local image file. It is clear what the tool does, but it never names or contrasts with the sibling tools image_ocr or image_decide, so an agent must infer the distinction.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by 'Ask ... a question about a local image file', and there is one concrete constraint ('Use an absolute path'). There is no explicit when-to-use / when-not guidance and no routing to image_ocr or image_decide.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

image_ocrA
Read-only

Read text and image-local bounding regions from an absolute local image path. query filters OCR text. Coordinates describe pixels, not desktop click targets.

NameRequiredDescriptionDefault
pathYes
limitNo
queryNo
cursorNo

TDQS

A3.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover the read-only, non-destructive, closed-world profile. The description adds genuinely useful behavioral context beyond them: the path must be absolute and local, and returned coordinates are pixel-space, not desktop click targets — a real disambiguation for an agent.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three short sentences, each carrying distinct information (what it reads, what query does, what coordinates mean), with the core action front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description usefully sketches what comes back (OCR text plus image-local bounding regions in pixel coordinates). However, the presence of limit and cursor signals a paginated result that the description never explains, leaving an agent without guidance on retrieving more than one page.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate and only partially does. It clarifies path (absolute local) and query (filters OCR text), but limit and cursor are never mentioned, leaving pagination behavior completely undocumented in both schema and description.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: reads OCR text and image-local bounding regions from an absolute local image path. This is clearly distinct from image_describe and image_decide without naming them, though explicit sibling differentiation is absent.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied — this is the tool for extracting text from an image — but there is no explicit when-to-use versus image_describe/image_decide, and no guidance on when to supply query versus reading all text.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 3 tool updatesv0.1.0
    • First observedimage_decide
    • First observedimage_describe
    • First observedimage_ocr
Deploy Server
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

A3.6/5.0

Scored across 3 tools

Disambiguation3/5

image_ocr is clearly separate (text extraction), but image_describe and image_decide both reduce to 'ask a model a question about an image' and differ mainly by answer format and backend model (Liquid CPU vs H2O). The descriptions help, but an agent must parse subtle wording to choose between free-text description and constrained boolean/choice/score answers.

Naming Consistency4/5

All three tools share a clean snake_case `image_` prefix, which makes them obviously part of one family. Minor deviation: `image_ocr` uses an acronym-noun while `image_describe`/`image_decide` use verbs, so the action pattern isn't perfectly uniform.

Tool Count4/5

Three tools is on the thin side but each covers a genuinely different modality (OCR with regions, free-text description, constrained decision/scoring). Nothing feels redundant, though the surface is small for a general vision server.

Completeness3/5

Core read-only vision operations are covered, but there is no image metadata/info tool, no object/localization tool beyond OCR regions, no multi-image comparison, and no preprocessing (crop/resize) step. Agents can partially work around these via describe/decide, but several natural vision tasks are dead ends.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Resources

Related MCP Connectors

Related MCP Servers