# windows-control

An MCP server that lets an AI actually operate Windows apps — select a window, read
what's on screen, click things by name, type, scroll, drag — plus a **WhatsApp layer**
that understands the app's UI (unread chats, contacts, conversations) instead of
returning a wall of OCR text.

Three rules shape the design:

- **No injection.** Nothing is loaded into the target process and no automation or
  accessibility provider is queried. The app is never touched from the inside.
- **Windows OCR for sight.** `Windows.Media.Ocr` via WinRT — offline, built into
  Windows, no third-party service.
- **Background by default (WhatsApp).** The app is captured with `PrintWindow` and
  clicked with posted messages, so it can sit fully covered behind whatever you're
  doing. Your cursor never moves and your focus is never stolen.

## Setup

```powershell
py -m pip install -r requirements.txt
py test_control.py             # generic: opens Notepad, types, OCRs, clicks a menu
py test_whatsapp.py            # WhatsApp: pixel heuristics + a live read-only run
py test_whatsapp.py --offline  # no WhatsApp needed
```

Copy `.mcp.json.example` to `.mcp.json` and set `cwd` to wherever you cloned
this, and Claude Code picks the server up in that folder. For Claude Desktop,
add the same block to `claude_desktop_config.json`:

```json
{
  "mcpServers": {
    "windows-control": {
      "command": "py",
      "args": ["-m", "windows_control.server"],
      "cwd": "C:\\path\\to\\windows-control-mcp"
    }
  }
}
```

## Foreground vs background

Two ways to drive an app. Both are real OS-level input; neither injects.

| | Foreground (`background=False`) | Background (`background=True`) |
| --- | --- | --- |
| See | screen capture of the window | `PrintWindow(PW_RENDERFULLCONTENT)` — works while fully covered |
| Click / scroll | `SendInput` — the cursor physically moves | `PostMessage` to the app's render widget — cursor untouched |
| Type | `SendInput` Unicode keystrokes | `WM_CHAR` posted to the render widget |
| Disturbs you | yes — takes over screen and cursor | no |
| Works with | every app | Chromium/Electron/WebView2 apps (WhatsApp, Slack, Discord, VS Code) |

**Everything works in the background, including sending.** Reading, opening
chats, scrolling, searching and sending a message all leave your focus, cursor
and window order untouched — verified by watching the foreground and z-order
continuously through a real send.

Typing needs one specific detail: post **`WM_CHAR` alone**. Wrapping it in the
`WM_KEYDOWN`/`WM_KEYUP` pair looks more faithful to a real keyboard, but
Chromium derives its own character from the key event *and* processes yours, so
every letter arrives two or three times (`abcd` → `aabbccdd`). A bare
`WM_KEYDOWN` alone does nothing.

If posted typing is ever refused, the field is cleared and the typing is redone
by briefly borrowing the foreground, then handing it straight back — so it
degrades instead of half-typing. Set **`WHATSAPP_NEVER_FOCUS=1`** to forbid even
that, making it impossible for anything to bring WhatsApp forward.

### Sending safely

`whatsapp_send_message` **reads the text back out of the composer before pressing
Enter**, so a half-typed or mistyped message is never sent, and it refuses
outright if the open chat isn't the one you asked for. Afterwards it confirms
from the screen: `sent` (the box returned to its placeholder) and
`confirmed_in_chat` (the message is visible in the conversation).

A covered Chromium window throttles repainting, so what you type can take several
seconds to appear in a capture — verification polls for up to 20s rather than
judging on the first look. Expect a send to take ~25s.

The window must not be **minimised** for background capture — it can be behind
anything, but a minimised window has nothing to render.

## WhatsApp tools

These return structured data, not raw OCR dumps.

| Tool | What it does |
| --- | --- |
| `whatsapp_overview` | The chat list: names, previews, times, unread badges, total unread |
| `whatsapp_unread(scan=…)` | Chats with unread messages — three levels of thoroughness |
| `whatsapp_watch_unread` | Waits for a *new* unread message and reports who it's from |
| `whatsapp_open_chat(name)` | Opens a contact/group — scrolls the list to find it, then clicks (background) |
| `whatsapp_search(query)` | WhatsApp's own search — typed in the background, verified |
| `whatsapp_read_chat(chat, scroll_back)` | Conversation content — messages, senders, times, history |
| `whatsapp_send_message(text, to)` | Sends — irreversible, confirm with the user first |
| `whatsapp_list_media` | Images, videos and documents visible in the open chat |
| `whatsapp_download_media(index)` | Saves an attachment to disk |
| `whatsapp_scroll(pane, amount)` | Scroll `"chats"` or `"conversation"` |
| `whatsapp_set_filter(name)` | Click the All / Unread / Favourites pill |

### Detecting unread messages

Unread is decided by a **pixel test** — the green badge disc on the row — not by
OCR, so it works even when the number inside the badge can't be read. Three
scan modes trade speed for completeness, and all three stay in the background:

| `scan` | What it does | Cost | Misses anything? |
| --- | --- | --- | --- |
| `"visible"` (default) | Badges in the on-screen part of the list | ~4s, one capture | Chats below the fold — says so via `complete: false` and a note |
| `"scroll"` | Scrolls the whole list collecting badges, scrolls back | ~5–10s | No |
| `"filter"` | Clicks WhatsApp's own Unread filter, reads it, restores your filter | ~20s | No — the definitive list |

`"filter"` changes UI state, so it remembers which pill was selected (detected by
its solid fill, since OCR drops a word as short as "All"), clicks it back, and
**verifies** the restore actually took, retrying if not. If it still can't, the
result says so rather than leaving your list stuck on a filter you didn't pick.

`whatsapp_watch_unread(timeout_seconds, poll_seconds)` takes a baseline first, so
chats that are *already* unread don't trigger it — only new arrivals do. It polls
in the background, so you can keep working while it waits.

```
whatsapp_unread()                          # fast look: who's waiting
whatsapp_unread(scan="filter")             # definitive: every unread chat
whatsapp_watch_unread(timeout_seconds=600) # tell me when someone messages
```

### Reading chat content

`whatsapp_read_chat` returns the conversation as structured messages — **one per
bubble**, so a long multi-line message stays whole instead of shattering into a
line-per-entry:

```json
{ "text": "Running late\nStarting in 20 minutes\nSee you there",
  "sender": "them", "sender_name": "Alex Rivera",
  "sender_phone": "+1 555 0100", "time": "1:42am", "kind": "message" }
```

- `sender` — `you` / `them` from the bubble colour, or `system`
- `sender_name` — who wrote it in a group chat (WhatsApp draws it in colour)
- `kind` — `message`, or `system` for join notices and privacy changes
- `transcript` — the whole thing rendered ready to quote

```
whatsapp_read_chat()                          # the open chat
whatsapp_read_chat(chat="Mom")                # open it first, then read
whatsapp_read_chat(scroll_back=3)             # include older history
```

`chat=` finds the conversation by scrolling the list, so it stays background.
`scroll_back=N` walks up N screens and stitches them into one chronological
transcript, then returns the view to the newest messages.

Bubbles are found by **colour continuity**, not by gaps between lines — the
bubble's background runs unbroken through the blank lines inside a long message,
whereas the wallpaper shows through between two separate messages. That one
distinction is what keeps a multi-line message intact and stops two consecutive
messages merging.

### Downloading attachments

```
whatsapp_list_media()            # what's attached in this chat
whatsapp_download_media()        # save the newest one           [background]
whatsapp_download_media(index=0, folder="C:\\keep")
```

Photos and videos contain **no text**, so OCR cannot see them at all — they're
found as blocks of high colour variance, which is what separates a photograph
from a flat message bubble and from the faint wallpaper. Documents are found by
their filename, which *is* text. A video is told apart from a photo by the
duration burnt into its thumbnail — matched as a bare `0:42`, since a bubble
timestamp looks nearly identical but carries am/pm.

Downloading right-clicks the attachment, picks WhatsApp's own *Save as*, and
fills in the Windows save dialog — all without focusing or raising WhatsApp.
Files go to `~/Downloads/whatsapp` unless you pass `folder` or set
`WHATSAPP_DOWNLOAD_DIR`.

Two traps the implementation handles, both of which silently produce *no file*:

- **`WM_SETTEXT` on the filename box does nothing useful.** It changes the text
  you can see but not the model the picker reads, so it commits the original
  name. The field has to be edited the way an edit control expects
  (`EM_SETSEL` + `EM_REPLACESEL`).
- **The name you ask for is not the name you get.** The picker enforces the
  source file's extension, so `photo.png` on a JPEG is written as
  `photo.png.jpeg`. The saved file is located by diffing the directory, and the
  returned `path` is authoritative — never assume the name you requested.

Only attachments **on screen** can be downloaded; scroll to older ones first.

### How it understands the UI

No hardcoded coordinates — every cue is derived from the pixels, so it survives
window resizing and theme changes:

- **Panels** — the nav rail, chat list and conversation pane have different
  background colours, so the dividers are found by scanning for column colour
  transitions (with a proportional fallback if the read looks implausible).
- **Chat name vs preview** — the name is the highest-contrast text in a row,
  measured against *that row's own* background so the highlighted (open) row
  still parses correctly. Works in light and dark themes.
- **Unread** — the green badge is found as a solid, round, right-aligned colour
  blob (#21C063/#25D366), which is why a green timestamp or a green group avatar
  doesn't fool it.
- **Not-a-chat rows** — the end-to-end-encryption notice and similar banners are
  dropped, and icons that OCR reads as a stray one-character word (the mute bell
  becoming "Family O") are stripped using the gap between word boxes.
- **Sender names** — drawn in colour, so they're found by saturation rather than
  by position. Links are coloured too, so a name only counts when plain text
  follows it — otherwise a green URL would be read as the sender.
- **Avatars** — a group avatar's initial OCRs as a one-letter line in the gutter
  left of the bubbles; it's dropped rather than reported as a message.
- **You vs them** — sent bubbles are green; the bubble background colour beside
  each message decides, falling back to alignment.

### Example

```
whatsapp_overview()                       # who's messaged, what's unread   [background]
whatsapp_open_chat("Mom")                 # scrolls the list, clicks it     [background]
whatsapp_read_chat()                      # transcript, tagged you/them     [background]
whatsapp_send_message("On my way 🙂")     # verified, then sent    [background]
```

## Generic tools

**Choosing an app:** `list_windows`, `focus_window`, `active_window`,
`window_action` (maximize/minimize/restore/close), `launch_app`

**Seeing:** `read_screen`, `find_text`, `wait_for_text`, `screenshot` —
each takes `background=True` with a `window`

**Acting:** `click_text`, `click`, `type_text`, `press_keys`, `move_mouse`,
`drag`, `scroll`, `cursor_position` — `click`, `click_text` and `scroll` take
`background=True`

Coordinates are absolute physical screen pixels across all monitors (the process
is per-monitor DPI-aware), so a `center_x/center_y` from `find_text` goes
straight into `click`.

## Known limits

- **OCR sees text, not icons.** Unlabelled icon buttons have nothing to match —
  screenshot and click coordinates for those.
- **Per-chat unread *counts* are best-effort.** Windows OCR is a line recogniser
  and usually refuses a lone digit, so a chat may come back `has_unread: true`
  with `unread: null`. It never guesses a number. *Which* chats are unread is
  reliable (a pixel test), and how many chats are unread comes from the
  "Unread N" pill, which is ordinary text and reads reliably.
- **Media has no text to read.** Images, voice notes and stickers come back as
  their caption or not at all — OCR reads pixels of text, not pictures. Use
  `whatsapp_download_media` to get the file itself.
- **OCR costs ~1–4s per call**, dominated by recognition, not by starting
  PowerShell: a long-lived OCR host process is reused across calls (starting one
  per call added seconds every time). Each read parses a single screenshot, so
  prefer `whatsapp_overview` / `whatsapp_read_chat` over repeated `find_text`.
- **Opening a chat marks it read**, exactly as it would if you clicked it.
- **Background input is app-dependent.** It works on Chromium/WebView2 apps;
  native Win32 apps may ignore posted clicks. Verify with a background
  screenshot, or fall back to `background=False`.
- **Foreground mode shares your mouse.** Don't type while it works.
- `WHATSAPP_NEVER_FOCUS=1` forbids ever raising WhatsApp (typing then errors).
  `WHATSAPP_READONLY=1` disables sending.
  `WINDOWS_CONTROL_BLOCKLIST=banking,keepass` refuses control of matching windows.
  `WINDOWS_CONTROL_OCR_LANG=en-US` pins the OCR language.

## Layout

```
windows_control/
  server.py     MCP tool definitions (25 tools)
  whatsapp.py   WhatsApp UI understanding: panels, rows, badges, messages
  bginput.py    background capture (PrintWindow) + posted clicks/scroll/typing
  filedialog.py drives the Windows save picker without focusing it
  wininput.py   foreground SendInput mouse + Unicode keyboard
  winapi.py     window enumeration, verified focus, focus borrowing, DPI
  screen.py     capture, OCR (persistent host), text -> coordinate matching
  ocr.ps1       Windows.Media.Ocr via WinRT (one-shot or -Server mode)
test_control.py    generic end-to-end test
test_whatsapp.py   WhatsApp tests (offline heuristics + live background run)
```