screencye — eyes for text-only LLMs

Decode a screenshot into exact structured text so any text-only model can "see" your UI — words, coordinates, sizes, colors, spacing. Pure-code CV + OCR. Zero VRAM. Deterministic.

Give a screenshot to a text-only model (DeepSeek, a local model, Claude Code, Hermes, etc.) and it can now reason about exact positions instead of hallucinating them — because the screenshot was decoded into a transcript with precise measurements.

┌ CARD "Welcome back ..." at (431,197) 418×406 · fill #ffffff
  │ TEXT "Welcome back" at (556,240) 168×19 · #111827
  ┌ INPUT "you@example.com" at (468,341) 344×40 · fill #ffffff
  ┌ BUTTON "Log in" at (467,517) 346×46 · fill #2563eb · text #ffffff

Here's a text-only agent (Hermes) using both tools — decode_screenshot + describe_screenshot — to answer "what do you see?":

Why it exists

• Text-only models can't see screenshots — and describing a misaligned button in words is error-prone.

• Vision models steal VRAM — a local vision encoder (like Gemma's --mmproj) lives in GPU memory even when idle, squeezing the text model.

• screencye runs on CPU — the decode is pure code + PaddleOCR via ONNX Runtime. No GPU, no network, no vision model in the reading path. All your VRAM stays with your text model. (The optional describe_screenshot tool uses a tiny on-CPU MobileCLIP2-S2 classifier — still zero VRAM.)

Related MCP server: uitars-mcp

Skip the vision encoder — save the VRAM

Running a local vision-language model in llama-server (Qwen-VL, Gemma 3, LLaVA, MiniCPM-V)? That --mmproj flag is its vision encoder — a separate projector file (~0.8–1.1 GB) sitting in VRAM on top of the LLM, even when you're only reading text.

For reading screens you don't need a vision model — you need the information in the image. screencye turns any screenshot into exact text (words, coordinates, colors, spacing) with a ~21 MB on-device engine and deterministic pixel analysis. Zero VRAM. Runs on CPU.

Drop --mmproj, run the model text-only, and let screencye do the looking:

Same ability to read a UI at a fraction of the memory — and because the decode is exhaustive and deterministic, nothing is silently missed the way a vision encoder's selective attention can skip details.

If your job is understanding arbitrary images — a photo's subject, a chart's trend — keep the vision model. screencye is for screens: exact, complete, and nearly free to run.

How it works (no AI in the decode)

1. OCR — PaddleOCR v5 mobile (ONNX Runtime, ~21 MB) reads every word with a bounding box + confidence.

2. Layout — pure pixel code: Sobel edges, color-quantized flood fill, connected components → finds buttons, inputs, cards.

3. Inference — geometric heuristics classify each box (centered text in a bordered box = button, etc.).

4. Transcript — computed coordinates, spacing, alignment, colors; rendered as a nested tree in reading order.

Deterministic: same screenshot → byte-identical transcript, every time.

Install

CLI (any agent or script)

npm install -g github:veloce-ai-idm/deepsee
screencye /path/to/screenshot.png

MCP server (Claude Code, Hermes, etc.)

Add to your agent's MCP config (claude mcp add or the client's MCP settings):

{
  "mcpServers": {
    "screencye": {
      "command": "screencye-mcp",
      "args": []
    }
  }
}

Then any agent can call the decode_screenshot tool with a file path and get the transcript.

Model files

All models ship in the repo's models/ folder (~91 MB total, each file under GitHub's 100 MB limit):

• det_infer.onnx, rec_infer.onnx, ppocrv5_dict.txt — PaddleOCR v5 (reads every word)

• mobileclip-vision.onnx (fp16, 73 MB) + mobileclip-labels.json — MobileCLIP2-S2 semantic tagger

Resolution order:

1. SCREENCYE_MODEL_DIR env var (explicit override)

2. <install>/models/ (ships with the package)

The label list lives in scripts/build_labels.py (one-time build: tokenizes labels and runs the MobileCLIP text encoder; needs the text ONNX, ~250 MB, from RuteNL/MobileCLIP2-S2-OpenCLIP-ONNX). The browser app (deepsee.veloceidm.com) serves the same fp16 model split into two ~37 MB parts — the IONOS host caps files at 50 MB, so it's chunked and reassembled at load time, not re-quantized.

Bigger models (S3/S4, higher zero-shot accuracy) are NOT bundled — their fp16 exports exceed GitHub's 100 MB/file limit, so they can't ship in this repo. Power users can point SCREENCYE_MODEL_DIR at an S3/S4 mobileclip-vision.onnx (from RuteNL/MobileCLIP2-S3-OpenCLIP-ONNX or S4) for a ~3–5% zero-shot accuracy boost.

Tools

describe_screenshot classifies against ~96 broad labels (UI types, games, photos, documents, charts, code, media, abstract). If no label clears the confidence threshold it appends a LOW CONFIDENCE warning instead of forcing a guess — so a blind model isn't misled while debugging. The label list lives in scripts/build_labels.py.

Privacy

Everything runs locally. The screenshot never leaves the machine — no API calls, no data egress.

Test

npm test        # parity + structure + determinism on golden screenshots
node test/mcp-handshake-test.mjs   # full MCP handshake

Files

Roadmap

• decode_screenshot_base64 — pass image bytes directly (no temp file needed)

• screenshot capture helper

Powered by VELOCE AI Accelerator · ONNX Runtime · PaddleOCR