# ScreenMind

Structured Windows screen maps for AI agents, exposed to OpenCode over MCP.

The agent queries **buttons, text, fields, positions, and available actions** without analyzing an image. When it needs pixels, it asks for a downscaled screenshot. It can click with a humanized mouse path, type, and chain steps.

```
OpenCode  ──MCP stdio──>  screenmind serve
                              ├── UI Automation (element tree, stable IDs)
                              ├── Watcher (focus + UI diffs in the background)
                              ├── Win32 (mouse/keyboard, DPI, windows)
                              └── On-demand capture (downscaled JPEG/PNG) + OCR
```

## Requirements

- Windows 10/11 with an unlocked interactive session (does not work on the secure desktop).
- Python 3.10+ (tested with 3.11.9).
- OpenCode with MCP support.
- Optional: Tesseract OCR (`C:\Program Files\Tesseract-OCR\tesseract.exe`) for text drawn on canvas/images.

## Installation

```powershell
cd C:\path\to\screenmind
py -3.11 -m venv .venv
.\.venv\Scripts\python.exe -m pip install -e .
.\.venv\Scripts\python.exe -m screenmind doctor
```

If everything is `OK`, register the server in OpenCode:

```powershell
.\.venv\Scripts\python.exe -m screenmind install-opencode --global
```

The command backs up `opencode.json`, preserves the other keys, and adds `mcp.servers.screen`. Restart OpenCode. To check:

```powershell
opencode mcp list
```

Expected output: `✓ screen connected` (or `pending` until you restart).

Manual alternative: copy the block from `examples/opencode.json` into your `opencode.json` and adjust the Python path.

## MCP tools

In OpenCode they appear as `screen_*` (or `tools.screen.*` in Code Mode).

| Tool | What it does |
|---|---|
| `status` | Active window, monitors, cursor, watcher, OCR, paths, and capabilities. |
| `map` | Map of the active window/specific window/desktop. Returns `id`, `role`, `name`, `value`, `rect`, `center`, `actions`. Filters by `query` and `roles`. |
| `inspect` | Details for one `id`: UIA patterns, children, window, internal key. |
| `click` | Click by `id`, visible text, or `x,y`; button, double click, and `prefer_invoke`. |
| `type` | Type into the target/focus, with `clear`, `submit`, human pacing, and paste for long text. |
| `key` | Shortcuts: `enter`, `ctrl+s`, `alt+f4`, `f5`, etc. |
| `scroll` | Scroll over the target/coordinates/window center. |
| `act` | Sequence of up to 25 steps in one call (saves round-trips). |
| `wait_for` | Wait for an element to appear/disappear/change value/become enabled. |
| `windows` | List windows and apply `focus`, `maximize`, `minimize`, `restore`. |
| `events` | Recent events: focus changes and UI diffs (what appeared/disappeared). |
| `shot` | On-demand downscaled screenshot with a `coordinate_hint` for pixel → screen. |
| `ocr` | Optional Tesseract OCR in a region, with boxes in screen coordinates. |
| `clipboard` | Read/write clipboard text (avoid sensitive data). |

### Recommended agent flow

1. `status` — confirm the desktop is available and see the active window.
2. `map(query="...")` — locate the element and keep its `id`.
3. `click(target=<id>)` / `type(target=<id>, text="...")`.
4. `map(refresh=true)` or `wait_for(...)` — verify the change.
5. `shot(max_width=800, quality=60)` — only when the accessibility tree does not cover the content.

## How it works

### Map and stable IDs

The scan walks the UI Automation tree with depth and node limits. Each element gets a short `id`, reused across scans through the Windows `RuntimeId`. The registry keeps the COM control alive for up to 15 minutes (configurable); if the element disappears, the tool asks for a new `map`.

- Search text ignores accents and letter case.
- Actionable elements are ranked higher.
- Accessibility duplicates (same role/name/rect) are dropped.
- `actions` lists the available patterns: `invoke`, `value`, `toggle`, `expand`, `select`, `range`.
- The watcher keeps a shallow snapshot of the active window (`depth=12`) and publishes diffs; the full `map` uses `depth=14` and a 3 s cache.
- By default the map returns only visible elements. `include_offscreen=true` also walks content outside the visible area (slower and more tokens).

### Humanized click

The cursor follows a cubic Bézier curve with speed control (acceleration/deceleration), decreasing jitter, a slight overshoot, and a final correction. Duration grows with distance (0.16–0.62 s). `natural=false` teleports to the point.

The click point comes from `GetClickablePoint` when valid; otherwise from the center of the rectangle. `prefer_invoke=true` uses the UIA Invoke pattern and does not move the mouse.

### Economical screenshots

- `shot` captures the active window, the full screen, or a region `[x,y,w,h]`.
- It resizes with LANCZOS to `max_width` (default 1152) and compresses to JPEG (quality 72) or PNG.
- It returns the MCP image plus metadata and the conversion formula:

```
screen_x = region.x + image_pixel_x / scale
screen_y = region.y + image_pixel_y / scale
```

Measured on this machine (1366×768): a window screenshot at 640 px ≈ 16 KB; 342 px ≈ 9 KB; capture + compression ≈ 80 ms. The agent only pays image tokens when it asks.

### Input and safety

- `SendInput` for mouse/keyboard; text via `KEYEVENTF_UNICODE` (layout-independent).
- Per-monitor DPI: physical coordinates match UI Automation and capture.
- The top-left corner (virtual screen origin) is an **abort zone**: actions are refused there.
- `win+l` is blocked; clearing (`clear=true`) requires an explicit target in `Edit`/`ComboBox`/`Spinner`, uses ValuePattern when available, and confirms the field is empty.
- Actions are serialized; the watcher does not interfere with clicks.
- Measured on this machine: the watcher runs every ~2 s (5 scans in 10 s) using ~9% of one core; most of the time is COM waiting, not CPU.
- Action log at `%LOCALAPPDATA%\screenmind\actions.jsonl` (without the typed text).
- Works only in the interactive session; elevated apps (UAC) do not accept input from a non-elevated process.

## CLI

```powershell
python -m screenmind doctor [--json]         # real diagnostics for UIA, capture, input, and OCR
python -m screenmind map --limit 20          # readable map in the terminal
python -m screenmind map --query "Save" --json
python -m screenmind shot --width 800 --out print.jpg
python -m screenmind watch --seconds 30      # watcher events
python -m screenmind config --init           # create config.json with defaults
python -m screenmind serve                   # MCP stdio server
```

## Configuration

File: `%LOCALAPPDATA%\screenmind\config.json` (create it with `config --init`). Main keys:

| Key | Default | Effect |
|---|---|---|
| `scan_depth` | 14 | Depth of the full map. |
| `scan_max_nodes` | 1500 | Node budget per scan. |
| `watcher_enabled` | true | Background watcher. |
| `watcher_interval_s` | 1.2 | Minimum interval between watcher scans. |
| `watcher_depth` | 12 | Watcher depth (cheaper). |
| `map_limit` | 60 | Elements returned by default. |
| `map_cache_s` | 3.0 | Reuse of the last map. |
| `shot_max_width` | 1152 | Default maximum capture width. |
| `shot_quality` | 72 | JPEG quality. |
| `allow_input` | true | `false` puts actions in dry-run mode. |
| `type_char_s` | 0.035 | Base delay for human typing. |
| `tesseract_cmd` | auto | Path to Tesseract. |

Equivalent environment variables: `SCREENMIND_WATCHER`, `SCREENMIND_ALLOW_INPUT`, `SCREENMIND_MAX_NODES`, `SCREENMIND_MAP_LIMIT`, `SCREENMIND_SHOT_WIDTH`, `SCREENMIND_LOG_LEVEL`, `SCREENMIND_DATA_DIR`.

## Tests

```powershell
.\.venv\Scripts\python.exe -m pytest -q
.\.venv\Scripts\python.exe tools\mcp_smoke.py   # MCP stdio + image + error
.\.venv\Scripts\python.exe tools\ui_e2e.py      # real WinForms window: map, click, type, act, wait_for, shot, OCR
```

`ui_e2e.py` opens a fixture window, clicks `Click Me`, types `hello screenmind 123`, clicks `Snapshot`, checks the fixture log, captures the window, and runs OCR. It restores the cursor position at the end.

## Known limitations

- Coverage depends on the app exposing UI Automation. Canvas, games, VNC/RDP, and custom-rendered apps may require screenshot + OCR or coordinates.
- Chrome exposes a good tree; on very deep pages a scan takes ~0.5–1.3 s. Use `depth=10..12` for faster, shallower maps and `query` to reduce tokens.
- Default OCR is English; install additional languages in Tesseract and use `lang="eng"`/another code as needed.
- The tool does not elevate privileges; administrator windows may refuse clicks.
- The watcher observes the active window. For other apps, use `windows(action="focus")` and then `map`.

## Structure

```
screenmind/
  screenmind/
    cli.py          # subcommands and OpenCode registration
    server.py       # MCP server (14 tools)
    service.py      # map, target resolution, actions, and safety
    uia.py          # UIA scan, ID registry, diff, and COM runtime
    watcher.py      # background watcher and event log
    humanize.py     # mouse path, typing, and scroll
    winapi.py       # DPI, windows, SendInput, clipboard
    screenshot.py   # capture, resize, and compression
    ocr.py          # optional Tesseract
    config.py       # settings and paths
  tools/            # MCP smoke test, WinForms fixture, and E2E test
  tests/            # unit tests
  examples/         # OpenCode configuration block
```

## License

MIT. See [LICENSE](LICENSE).