ScreenMind

Structured Windows screen maps for AI agents, exposed to OpenCode over MCP.

The agent queries buttons, text, fields, positions, and available actions without analyzing an image. When it needs pixels, it asks for a downscaled screenshot. It can click with a humanized mouse path, type, and chain steps.

OpenCode  ──MCP stdio──>  screenmind serve
                              ├── UI Automation (element tree, stable IDs)
                              ├── Watcher (focus + UI diffs in the background)
                              ├── Win32 (mouse/keyboard, DPI, windows)
                              └── On-demand capture (downscaled JPEG/PNG) + OCR

Requirements

• Windows 10/11 with an unlocked interactive session (does not work on the secure desktop).

• Python 3.10+ (tested with 3.11.9).

• OpenCode with MCP support.

• Optional: Tesseract OCR (C:\Program Files\Tesseract-OCR\tesseract.exe) for text drawn on canvas/images.

Related MCP server: Windows-MCP

Installation

cd C:\path\to\screenmind
py -3.11 -m venv .venv
.\.venv\Scripts\python.exe -m pip install -e .
.\.venv\Scripts\python.exe -m screenmind doctor

If everything is OK, register the server in OpenCode:

.\.venv\Scripts\python.exe -m screenmind install-opencode --global

The command backs up opencode.json, preserves the other keys, and adds mcp.servers.screen. Restart OpenCode. To check:

opencode mcp list

Expected output: ✓ screen connected (or pending until you restart).

Manual alternative: copy the block from examples/opencode.json into your opencode.json and adjust the Python path.

MCP tools

In OpenCode they appear as screen_* (or tools.screen.* in Code Mode).

Recommended agent flow

1. status — confirm the desktop is available and see the active window.

2. map(query="...") — locate the element and keep its id.

3. click(target=<id>) / type(target=<id>, text="...").

4. map(refresh=true) or wait_for(...) — verify the change.

5. shot(max_width=800, quality=60) — only when the accessibility tree does not cover the content.

How it works

Map and stable IDs

The scan walks the UI Automation tree with depth and node limits. Each element gets a short id, reused across scans through the Windows RuntimeId. The registry keeps the COM control alive for up to 15 minutes (configurable); if the element disappears, the tool asks for a new map.

• Search text ignores accents and letter case.

• Actionable elements are ranked higher.

• Accessibility duplicates (same role/name/rect) are dropped.

• actions lists the available patterns: invoke, value, toggle, expand, select, range.

• The watcher keeps a shallow snapshot of the active window (depth=12) and publishes diffs; the full map uses depth=14 and a 3 s cache.

• By default the map returns only visible elements. include_offscreen=true also walks content outside the visible area (slower and more tokens).

Humanized click

The cursor follows a cubic Bézier curve with speed control (acceleration/deceleration), decreasing jitter, a slight overshoot, and a final correction. Duration grows with distance (0.16–0.62 s). natural=false teleports to the point.

The click point comes from GetClickablePoint when valid; otherwise from the center of the rectangle. prefer_invoke=true uses the UIA Invoke pattern and does not move the mouse.

Economical screenshots

• shot captures the active window, the full screen, or a region [x,y,w,h].

• It resizes with LANCZOS to max_width (default 1152) and compresses to JPEG (quality 72) or PNG.

• It returns the MCP image plus metadata and the conversion formula:

screen_x = region.x + image_pixel_x / scale
screen_y = region.y + image_pixel_y / scale

Measured on this machine (1366×768): a window screenshot at 640 px ≈ 16 KB; 342 px ≈ 9 KB; capture + compression ≈ 80 ms. The agent only pays image tokens when it asks.

Input and safety

• SendInput for mouse/keyboard; text via KEYEVENTF_UNICODE (layout-independent).

• Per-monitor DPI: physical coordinates match UI Automation and capture.

• The top-left corner (virtual screen origin) is an abort zone: actions are refused there.

• win+l is blocked; clearing (clear=true) requires an explicit target in Edit/ComboBox/Spinner, uses ValuePattern when available, and confirms the field is empty.

• Actions are serialized; the watcher does not interfere with clicks.

• Measured on this machine: the watcher runs every ~2 s (5 scans in 10 s) using ~9% of one core; most of the time is COM waiting, not CPU.

• Action log at %LOCALAPPDATA%\screenmind\actions.jsonl (without the typed text).

• Works only in the interactive session; elevated apps (UAC) do not accept input from a non-elevated process.

CLI

python -m screenmind doctor [--json]         # real diagnostics for UIA, capture, input, and OCR
python -m screenmind map --limit 20          # readable map in the terminal
python -m screenmind map --query "Save" --json
python -m screenmind shot --width 800 --out print.jpg
python -m screenmind watch --seconds 30      # watcher events
python -m screenmind config --init           # create config.json with defaults
python -m screenmind serve                   # MCP stdio server

Configuration

File: %LOCALAPPDATA%\screenmind\config.json (create it with config --init). Main keys:

Equivalent environment variables: SCREENMIND_WATCHER, SCREENMIND_ALLOW_INPUT, SCREENMIND_MAX_NODES, SCREENMIND_MAP_LIMIT, SCREENMIND_SHOT_WIDTH, SCREENMIND_LOG_LEVEL, SCREENMIND_DATA_DIR.

Tests

.\.venv\Scripts\python.exe -m pytest -q
.\.venv\Scripts\python.exe tools\mcp_smoke.py   # MCP stdio + image + error
.\.venv\Scripts\python.exe tools\ui_e2e.py      # real WinForms window: map, click, type, act, wait_for, shot, OCR

ui_e2e.py opens a fixture window, clicks Click Me, types hello screenmind 123, clicks Snapshot, checks the fixture log, captures the window, and runs OCR. It restores the cursor position at the end.

Known limitations

• Coverage depends on the app exposing UI Automation. Canvas, games, VNC/RDP, and custom-rendered apps may require screenshot + OCR or coordinates.

• Chrome exposes a good tree; on very deep pages a scan takes ~0.5–1.3 s. Use depth=10..12 for faster, shallower maps and query to reduce tokens.

• Default OCR is English; install additional languages in Tesseract and use lang="eng"/another code as needed.

• The tool does not elevate privileges; administrator windows may refuse clicks.

• The watcher observes the active window. For other apps, use windows(action="focus") and then map.

Structure

screenmind/
  screenmind/
    cli.py          # subcommands and OpenCode registration
    server.py       # MCP server (14 tools)
    service.py      # map, target resolution, actions, and safety
    uia.py          # UIA scan, ID registry, diff, and COM runtime
    watcher.py      # background watcher and event log
    humanize.py     # mouse path, typing, and scroll
    winapi.py       # DPI, windows, SendInput, clipboard
    screenshot.py   # capture, resize, and compression
    ocr.py          # optional Tesseract
    config.py       # settings and paths
  tools/            # MCP smoke test, WinForms fixture, and E2E test
  tests/            # unit tests
  examples/         # OpenCode configuration block

License

MIT. See LICENSE.