Skip to main content
Glama

local-vision-mcp

License: MIT

A Bun-only MCP server that gives any calling model eyes: it takes an image (typically a browser screenshot from the browse MCP), sends it to a locally loaded LM Studio vision model, and returns the answer — or click coordinates ready for browse MCP's click({x, y}).

  • MCP over Streamable HTTP: http://localhost:8084/mcp (stateless) — or --stdio for clients that spawn on demand

  • Self-documenting vision test page (embedded in the binary): http://localhost:8084/ — see the smoke-test section below

  • Health check: http://localhost:8084/healthz

  • Vision backend: LM Studio (http://localhost:1234, lmstudio-community/ornith-1.0-35b-mlx by default)

Tools

Tool

Input

Output

describe_image

image or image_path, question?

Vision model's text answer

find_click_target

image_path, target, viewport_width/height?, origin_x/y? (for clips)

JSON {x, y, found, description} in viewport CSS px — feed straight to browse click

image_path is preferred for screenshots: the server reads the file itself, so megabytes of base64 never pass through the calling model's context.

Coordinate calibration: this vision model reports coordinates in an internal 1024x1024 frame regardless of image size; the server converts them to real viewport CSS pixels automatically (LMSTUDIO_VISION_CANVAS=1024, set 0 to disable).

Related MCP server: Atlas Vision MCP

Run

Running the binary with no arguments prints usage — nothing starts silently.

local-vision-mcp --help          # full usage, modes, env vars
local-vision-mcp serve           # HTTP server: test page + /mcp endpoint
local-vision-mcp serve --port 9   # override the port (default 8084, PORT env)
local-vision-mcp --stdio         # MCP over stdin/stdout — opencode spawns it
                                   # on demand; exits when the session ends
local-vision-mcp --version

# from source, same flags:
bun run src/index.ts serve

Build the standalone binary (Bun runtime embedded, no file deps):

bun run build          # -> ~/.bun/bin/local-vision-mcp

Register with opencode

Recommended: stdio (spawn on demand, no daemon):

opencode mcp add local-vision --global -- ~/.bun/bin/local-vision-mcp --stdio

Config equivalent (note: opencode does not expand ~ in command entries — use the absolute path, e.g. /Users/you/.bun/bin/...):

{
  "mcp": {
    "servers": {
      "local-vision": {
        "type": "local",
        "command": ["/Users/you/.bun/bin/local-vision-mcp", "--stdio"]
      }
    }
  }
}

Alternative: remote HTTP (requires the daemon running first):

local-vision-mcp &          # start the daemon
opencode mcp add local-vision --global --url http://localhost:8084/mcp

New-machine setup order matters for the HTTP variant: start the binary before opencode connects, or the MCP shows ECONNREFUSED until opencode service restart. The stdio variant has no such ordering problem.

CLI (manual testing)

The compiled binary has describe and find subcommands (same handlers as the MCP tools) — no server needed:

local-vision-mcp describe shot.jpg "what is in this image?"
local-vision-mcp find shot.jpg "the hamburger menu button"

Exit codes: 0 success, 1 model/tool error (message printed), 2 bad arguments (usage shown). Works from source too: bun run src/index.ts find shot.jpg "target".

The built-in vision loop smoke test

The test page embedded in the binary (http://localhost:8084/ in serve mode) is itself a self-documenting smoke test. Its controls are deliberately invisible to accessibility snapshots (no roles, no labels) — vision coordinates are the only way to click them — and the page explains the loop and tracks your progress:

local-vision-mcp serve     # then open http://localhost:8084/

Run the vision→click instructions below against the three targets listed on the page (hamburger → "Purge cache" in the sidebar → "Sign out" in the Account dropdown). Each card in the page's checklist lights up green when its target is actually clicked — all three green means the whole stack (binary → LM Studio → calibration → click) is healthy. Ideal after a model swap or on a new machine.

Instructions for a calling model without vision

Copy this into the system prompt of any agent that has the browse MCP and the local-vision MCP but cannot see images — or point the agent at docs/using-local-vision.md in this repo, which expands the same steps with recovery hints and error handling:

You cannot see images, so use local-vision to see for you:

1. browse mcp: emulate the viewport to 1280x800, deviceScaleFactor 2
2. browse mcp: screenshot to a file (never inline)
3. local-vision mcp: find_click_target(image_path, target, viewport 1280x800)
4. browse mcp: click the returned {x, y}

If the click misses, re-screenshot and retry with a more specific target.
Open closed menus before locating items inside them. To read a page instead
of clicking, call describe_image on the same screenshot with a question.

Tests

bun test          # 60 tests, ~350ms — fully hermetic (fake LM Studio, no browser)
bun check         # typecheck

Run these after any code change. Coverage:

  • test/unit.test.ts — pure logic: data-URL normalization, PNG/JPEG/WebP dimension parsing, JSON extraction from model prose, the full JSON-RPC core (initialize, tools/list, tools/call, errors, batches, notifications)

  • test/integration.test.ts — the vision tools against a fake LM Studio: prompt assembly, image/file loading, 1024-canvas coordinate calibration (viewport, image-dims, and clip-origin variants), stats, and every error path (no message output, unparsable replies, found:false, bad input)

  • test/cli.test.ts — the describe/find subcommands: argument validation, exit codes (2 usage / 1 error / 0 ok), output correctness

  • test/entry.test.ts — entry-point surface: bare invocation prints usage, serve [--port N], --version, unknown-option rejection

  • test/web.test.ts — the served routes (/, /healthz, 404), the embedded test page's contents, MCP-over-HTTP semantics (405 GET, 406 missing accept, 400 parse error, initialize headers, tools/call end-to-end), and the --stdio transport

No real LM Studio or browser is needed: test/setup.ts (bunfig preload) points LMSTUDIO_BASE_URL at per-file fake servers before any module loads — a lesson learned: module-level env reads happen at first import, so setting env in beforeAll is too late if another test file imported the module first.

Config (env)

`PORT` (8084)
`LMSTUDIO_BASE_URL` (http://localhost:1234)
`LMSTUDIO_MODEL` (lmstudio-community/ornith-1.0-35b-mlx)
`LMSTUDIO_API_TOKEN` (optional)
`LMSTUDIO_VISION_CANVAS` (1024, 0 disables)

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers