local-vision-mcp
by morf-bls
README.md
# local-vision-mcp
[](LICENSE)
A Bun-only MCP server that gives any calling model **eyes**: it takes an
image (typically a browser screenshot from the browse MCP), sends it to a
locally loaded LM Studio vision model, and returns the answer — or click
coordinates ready for browse MCP's `click({x, y})`.
- MCP over Streamable HTTP: `http://localhost:8084/mcp` (stateless) — or
`--stdio` for clients that spawn on demand
- Self-documenting vision test page (embedded in the binary):
`http://localhost:8084/` — see the smoke-test section below
- Health check: `http://localhost:8084/healthz`
- Vision backend: LM Studio (`http://localhost:1234`,
`lmstudio-community/ornith-1.0-35b-mlx` by default)
## Tools
| Tool | Input | Output |
|---|---|---|
| `describe_image` | `image` or `image_path`, `question?` | Vision model's text answer |
| `find_click_target` | `image_path`, `target`, `viewport_width/height?`, `origin_x/y?` (for clips) | JSON `{x, y, found, description}` in viewport CSS px — feed straight to browse `click` |
`image_path` is preferred for screenshots: the server reads the file itself,
so megabytes of base64 never pass through the calling model's context.
Coordinate calibration: this vision model reports coordinates in an internal
1024x1024 frame regardless of image size; the server converts them to real
viewport CSS pixels automatically (`LMSTUDIO_VISION_CANVAS=1024`, set `0` to
disable).
## Run
Running the binary with no arguments prints usage — nothing starts silently.
```sh
local-vision-mcp --help # full usage, modes, env vars
local-vision-mcp serve # HTTP server: test page + /mcp endpoint
local-vision-mcp serve --port 9 # override the port (default 8084, PORT env)
local-vision-mcp --stdio # MCP over stdin/stdout — opencode spawns it
# on demand; exits when the session ends
local-vision-mcp --version
# from source, same flags:
bun run src/index.ts serve
```
Build the standalone binary (Bun runtime embedded, no file deps):
```sh
bun run build # -> ~/.bun/bin/local-vision-mcp
```
## Register with opencode
**Recommended: stdio (spawn on demand, no daemon):**
```sh
opencode mcp add local-vision --global -- ~/.bun/bin/local-vision-mcp --stdio
```
Config equivalent (note: opencode does **not** expand `~` in `command`
entries — use the absolute path, e.g. `/Users/you/.bun/bin/...`):
```jsonc
{
"mcp": {
"servers": {
"local-vision": {
"type": "local",
"command": ["/Users/you/.bun/bin/local-vision-mcp", "--stdio"]
}
}
}
}
```
**Alternative: remote HTTP (requires the daemon running first):**
```sh
local-vision-mcp & # start the daemon
opencode mcp add local-vision --global --url http://localhost:8084/mcp
```
New-machine setup order matters for the HTTP variant: start the binary
*before* opencode connects, or the MCP shows `ECONNREFUSED` until
`opencode service restart`. The stdio variant has no such ordering problem.
## CLI (manual testing)
The compiled binary has `describe` and `find` subcommands (same handlers as
the MCP tools) — no server needed:
```sh
local-vision-mcp describe shot.jpg "what is in this image?"
local-vision-mcp find shot.jpg "the hamburger menu button"
```
Exit codes: `0` success, `1` model/tool error (message printed), `2` bad
arguments (usage shown). Works from source too:
`bun run src/index.ts find shot.jpg "target"`.
## The built-in vision loop smoke test
The test page embedded in the binary (`http://localhost:8084/` in serve mode)
is itself a self-documenting smoke test. Its controls are deliberately
invisible to accessibility snapshots (no roles, no labels) — vision
coordinates are the only way to click them — and the page explains the loop
and tracks your progress:
```sh
local-vision-mcp serve # then open http://localhost:8084/
```
Run the vision→click instructions below against the three targets listed on
the page (hamburger → "Purge cache" in the sidebar → "Sign out" in the Account
dropdown). Each card in the page's checklist lights up green when its target
is actually clicked — all three green means the whole stack (binary → LM
Studio → calibration → click) is healthy. Ideal after a model swap or on a
new machine.
## Instructions for a calling model without vision
Copy this into the system prompt of any agent that has the browse MCP and the
local-vision MCP but cannot see images — or point the agent at
`docs/using-local-vision.md` in this repo, which expands the same steps with
recovery hints and error handling:
```text
You cannot see images, so use local-vision to see for you:
1. browse mcp: emulate the viewport to 1280x800, deviceScaleFactor 2
2. browse mcp: screenshot to a file (never inline)
3. local-vision mcp: find_click_target(image_path, target, viewport 1280x800)
4. browse mcp: click the returned {x, y}
If the click misses, re-screenshot and retry with a more specific target.
Open closed menus before locating items inside them. To read a page instead
of clicking, call describe_image on the same screenshot with a question.
```
## Tests
```sh
bun test # 60 tests, ~350ms — fully hermetic (fake LM Studio, no browser)
bun check # typecheck
```
Run these after any code change. Coverage:
- `test/unit.test.ts` — pure logic: data-URL normalization, PNG/JPEG/WebP
dimension parsing, JSON extraction from model prose, the full JSON-RPC
core (initialize, tools/list, tools/call, errors, batches, notifications)
- `test/integration.test.ts` — the vision tools against a fake LM Studio:
prompt assembly, image/file loading, 1024-canvas coordinate calibration
(viewport, image-dims, and clip-origin variants), stats, and every error
path (no message output, unparsable replies, `found:false`, bad input)
- `test/cli.test.ts` — the `describe`/`find` subcommands: argument
validation, exit codes (2 usage / 1 error / 0 ok), output correctness
- `test/entry.test.ts` — entry-point surface: bare invocation prints usage,
`serve [--port N]`, `--version`, unknown-option rejection
- `test/web.test.ts` — the served routes (`/`, `/healthz`, 404), the embedded
test page's contents, MCP-over-HTTP
semantics (405 GET, 406 missing accept, 400 parse error, initialize
headers, tools/call end-to-end), and the `--stdio` transport
No real LM Studio or browser is needed: `test/setup.ts` (bunfig preload)
points `LMSTUDIO_BASE_URL` at per-file fake servers before any module
loads — a lesson learned: module-level env reads happen at first import, so
setting env in `beforeAll` is too late if another test file imported the
module first.
## Config (env)
```bash
`PORT` (8084)
`LMSTUDIO_BASE_URL` (http://localhost:1234)
`LMSTUDIO_MODEL` (lmstudio-community/ornith-1.0-35b-mlx)
`LMSTUDIO_API_TOKEN` (optional)
`LMSTUDIO_VISION_CANVAS` (1024, 0 disables)
```
This server cannot be deployed
Maintenance
ActivityMaintained
ResponsivenessNo recent community issues