windows-control

An MCP server that lets an AI actually operate Windows apps — select a window, read what's on screen, click things by name, type, scroll, drag — plus a WhatsApp layer that understands the app's UI (unread chats, contacts, conversations) instead of returning a wall of OCR text.

Three rules shape the design:

• No injection. Nothing is loaded into the target process and no automation or accessibility provider is queried. The app is never touched from the inside.

• Windows OCR for sight. Windows.Media.Ocr via WinRT — offline, built into Windows, no third-party service.

• Background by default (WhatsApp). The app is captured with PrintWindow and clicked with posted messages, so it can sit fully covered behind whatever you're doing. Your cursor never moves and your focus is never stolen.

Setup

py -m pip install -r requirements.txt
py test_control.py             # generic: opens Notepad, types, OCRs, clicks a menu
py test_whatsapp.py            # WhatsApp: pixel heuristics + a live read-only run
py test_whatsapp.py --offline  # no WhatsApp needed

Copy .mcp.json.example to .mcp.json and set cwd to wherever you cloned this, and Claude Code picks the server up in that folder. For Claude Desktop, add the same block to claude_desktop_config.json:

{
  "mcpServers": {
    "windows-control": {
      "command": "py",
      "args": ["-m", "windows_control.server"],
      "cwd": "C:\\path\\to\\windows-control-mcp"
    }
  }
}

Related MCP server: windows-computer-use-mcp

Foreground vs background

Two ways to drive an app. Both are real OS-level input; neither injects.

Everything works in the background, including sending. Reading, opening chats, scrolling, searching and sending a message all leave your focus, cursor and window order untouched — verified by watching the foreground and z-order continuously through a real send.

Typing needs one specific detail: post WM_CHAR alone. Wrapping it in the WM_KEYDOWN/WM_KEYUP pair looks more faithful to a real keyboard, but Chromium derives its own character from the key event and processes yours, so every letter arrives two or three times (abcd → aabbccdd). A bare WM_KEYDOWN alone does nothing.

If posted typing is ever refused, the field is cleared and the typing is redone by briefly borrowing the foreground, then handing it straight back — so it degrades instead of half-typing. Set WHATSAPP_NEVER_FOCUS=1 to forbid even that, making it impossible for anything to bring WhatsApp forward.

Sending safely

whatsapp_send_message reads the text back out of the composer before pressing Enter, so a half-typed or mistyped message is never sent, and it refuses outright if the open chat isn't the one you asked for. Afterwards it confirms from the screen: sent (the box returned to its placeholder) and confirmed_in_chat (the message is visible in the conversation).

A covered Chromium window throttles repainting, so what you type can take several seconds to appear in a capture — verification polls for up to 20s rather than judging on the first look. Expect a send to take ~25s.

The window must not be minimised for background capture — it can be behind anything, but a minimised window has nothing to render.

WhatsApp tools

These return structured data, not raw OCR dumps.

Detecting unread messages

Unread is decided by a pixel test — the green badge disc on the row — not by OCR, so it works even when the number inside the badge can't be read. Three scan modes trade speed for completeness, and all three stay in the background:

"filter" changes UI state, so it remembers which pill was selected (detected by its solid fill, since OCR drops a word as short as "All"), clicks it back, and verifies the restore actually took, retrying if not. If it still can't, the result says so rather than leaving your list stuck on a filter you didn't pick.

whatsapp_watch_unread(timeout_seconds, poll_seconds) takes a baseline first, so chats that are already unread don't trigger it — only new arrivals do. It polls in the background, so you can keep working while it waits.

whatsapp_unread()                          # fast look: who's waiting
whatsapp_unread(scan="filter")             # definitive: every unread chat
whatsapp_watch_unread(timeout_seconds=600) # tell me when someone messages

Reading chat content

whatsapp_read_chat returns the conversation as structured messages — one per bubble, so a long multi-line message stays whole instead of shattering into a line-per-entry:

{ "text": "Running late\nStarting in 20 minutes\nSee you there",
  "sender": "them", "sender_name": "Alex Rivera",
  "sender_phone": "+1 555 0100", "time": "1:42am", "kind": "message" }

• sender — you / them from the bubble colour, or system

• sender_name — who wrote it in a group chat (WhatsApp draws it in colour)

• kind — message, or system for join notices and privacy changes

• transcript — the whole thing rendered ready to quote

whatsapp_read_chat()                          # the open chat
whatsapp_read_chat(chat="Mom")                # open it first, then read
whatsapp_read_chat(scroll_back=3)             # include older history

chat= finds the conversation by scrolling the list, so it stays background. scroll_back=N walks up N screens and stitches them into one chronological transcript, then returns the view to the newest messages.

Bubbles are found by colour continuity, not by gaps between lines — the bubble's background runs unbroken through the blank lines inside a long message, whereas the wallpaper shows through between two separate messages. That one distinction is what keeps a multi-line message intact and stops two consecutive messages merging.

Downloading attachments

whatsapp_list_media()            # what's attached in this chat
whatsapp_download_media()        # save the newest one           [background]
whatsapp_download_media(index=0, folder="C:\\keep")

Photos and videos contain no text, so OCR cannot see them at all — they're found as blocks of high colour variance, which is what separates a photograph from a flat message bubble and from the faint wallpaper. Documents are found by their filename, which is text. A video is told apart from a photo by the duration burnt into its thumbnail — matched as a bare 0:42, since a bubble timestamp looks nearly identical but carries am/pm.

Downloading right-clicks the attachment, picks WhatsApp's own Save as, and fills in the Windows save dialog — all without focusing or raising WhatsApp. Files go to ~/Downloads/whatsapp unless you pass folder or set WHATSAPP_DOWNLOAD_DIR.

Two traps the implementation handles, both of which silently produce no file:

• WM_SETTEXT on the filename box does nothing useful. It changes the text you can see but not the model the picker reads, so it commits the original name. The field has to be edited the way an edit control expects (EM_SETSEL + EM_REPLACESEL).

• The name you ask for is not the name you get. The picker enforces the source file's extension, so photo.png on a JPEG is written as photo.png.jpeg. The saved file is located by diffing the directory, and the returned path is authoritative — never assume the name you requested.

Only attachments on screen can be downloaded; scroll to older ones first.

How it understands the UI

No hardcoded coordinates — every cue is derived from the pixels, so it survives window resizing and theme changes:

• Panels — the nav rail, chat list and conversation pane have different background colours, so the dividers are found by scanning for column colour transitions (with a proportional fallback if the read looks implausible).

• Chat name vs preview — the name is the highest-contrast text in a row, measured against that row's own background so the highlighted (open) row still parses correctly. Works in light and dark themes.

• Unread — the green badge is found as a solid, round, right-aligned colour blob (#21C063/#25D366), which is why a green timestamp or a green group avatar doesn't fool it.

• Not-a-chat rows — the end-to-end-encryption notice and similar banners are dropped, and icons that OCR reads as a stray one-character word (the mute bell becoming "Family O") are stripped using the gap between word boxes.

• Sender names — drawn in colour, so they're found by saturation rather than by position. Links are coloured too, so a name only counts when plain text follows it — otherwise a green URL would be read as the sender.

• Avatars — a group avatar's initial OCRs as a one-letter line in the gutter left of the bubbles; it's dropped rather than reported as a message.

• You vs them — sent bubbles are green; the bubble background colour beside each message decides, falling back to alignment.

Example

whatsapp_overview()                       # who's messaged, what's unread   [background]
whatsapp_open_chat("Mom")                 # scrolls the list, clicks it     [background]
whatsapp_read_chat()                      # transcript, tagged you/them     [background]
whatsapp_send_message("On my way 🙂")     # verified, then sent    [background]

Generic tools

Choosing an app: list_windows, focus_window, active_window, window_action (maximize/minimize/restore/close), launch_app

Seeing: read_screen, find_text, wait_for_text, screenshot — each takes background=True with a window

Acting: click_text, click, type_text, press_keys, move_mouse, drag, scroll, cursor_position — click, click_text and scroll take background=True

Coordinates are absolute physical screen pixels across all monitors (the process is per-monitor DPI-aware), so a center_x/center_y from find_text goes straight into click.

Known limits

• OCR sees text, not icons. Unlabelled icon buttons have nothing to match — screenshot and click coordinates for those.

• Per-chat unread counts are best-effort. Windows OCR is a line recogniser and usually refuses a lone digit, so a chat may come back has_unread: true with unread: null. It never guesses a number. Which chats are unread is reliable (a pixel test), and how many chats are unread comes from the "Unread N" pill, which is ordinary text and reads reliably.

• Media has no text to read. Images, voice notes and stickers come back as their caption or not at all — OCR reads pixels of text, not pictures. Use whatsapp_download_media to get the file itself.

• OCR costs ~1–4s per call, dominated by recognition, not by starting PowerShell: a long-lived OCR host process is reused across calls (starting one per call added seconds every time). Each read parses a single screenshot, so prefer whatsapp_overview / whatsapp_read_chat over repeated find_text.

• Opening a chat marks it read, exactly as it would if you clicked it.

• Background input is app-dependent. It works on Chromium/WebView2 apps; native Win32 apps may ignore posted clicks. Verify with a background screenshot, or fall back to background=False.

• Foreground mode shares your mouse. Don't type while it works.

• WHATSAPP_NEVER_FOCUS=1 forbids ever raising WhatsApp (typing then errors). WHATSAPP_READONLY=1 disables sending. WINDOWS_CONTROL_BLOCKLIST=banking,keepass refuses control of matching windows. WINDOWS_CONTROL_OCR_LANG=en-US pins the OCR language.

Layout

windows_control/
  server.py     MCP tool definitions (25 tools)
  whatsapp.py   WhatsApp UI understanding: panels, rows, badges, messages
  bginput.py    background capture (PrintWindow) + posted clicks/scroll/typing
  filedialog.py drives the Windows save picker without focusing it
  wininput.py   foreground SendInput mouse + Unicode keyboard
  winapi.py     window enumeration, verified focus, focus borrowing, DPI
  screen.py     capture, OCR (persistent host), text -> coordinate matching
  ocr.ps1       Windows.Media.Ocr via WinRT (one-shot or -Server mode)
test_control.py    generic end-to-end test
test_whatsapp.py   WhatsApp tests (offline heuristics + live background run)