winauto-mcp

MCP server that lets an LLM see and drive any running Windows app: pick a window, screenshot it, read its UI Automation element tree (buttons/menus/ inputs with coordinates), click, type, and press keys. A transparent green outline tracks whichever window is currently attached; red boxes can be drawn on demand to highlight specific elements.

Windows-only. Built and tested against Python 3.12 on Windows 11.

Design

• Coordinates: every coordinate a tool accepts or returns (click x/y, element rects, highlight rects) is relative to the attached window's client area — the same pixel space as the image capture_screen returns. The server does all client-rect <-> screen-rect translation internally.

• Hybrid inspection: capture_screen returns both an image and a text summary built from the Windows UI Automation (UIA) tree. For native apps (browsers, Win32/WPF/UWP apps) the UIA summary is usually enough to act on by name. For canvas-rendered apps (games, custom-drawn UIs) the UIA tree comes back empty — the calling LLM should read the screenshot directly and pass pixel coordinates to click.

• Single active target: one attached window at a time (attach_window replaces whatever was attached before).

• Input simulation: uses the Win32 SendInput API directly (not pyautogui), with KEYEVENTF_UNICODE for typing — so arbitrary Unicode text (Thai included) types correctly without keycode mapping.

• DPI awareness: the server calls SetProcessDpiAwarenessContext (with fallbacks) at startup, before any other Win32/GDI call. Without this, an unaware process gets its coordinates silently scaled to 96 DPI while mss captures true physical pixels — on any scaled display (the common case) click coordinates read off a screenshot would land in the wrong place.

• Foreground-lock workaround: attach_window uses AttachThreadInput to temporarily join input queues with the current foreground window's thread before calling SetForegroundWindow — otherwise Windows silently ignores that call when it comes from a background/automated process (this server).

• Act on the current screen, not a remembered one: click and drag compare the area around their target against the frame the caller was actually shown covering that spot. If it changed, nothing is clicked — the call returns a refusal plus the window as it looks now, and re-issuing then goes through. This is the guard against the most common failure in a long run: deciding where to click, spending a few turns elsewhere, and clicking a coordinate the app has since moved on from. force=true skips it.

• "Seen" is per region, not per window: the server keeps the last few views the caller was shown — each with the rectangle it covered — rather than one whole-window frame. So looking at a 200x60 toolbar crop lets you click inside that toolbar and nothing else: a coordinate in a part of the window you have not looked at this run is refused as coming from memory. Without this, one partial capture would launder every stale coordinate on screen into "already checked".

• Look at part of a window, not all of it: capture_region(x1,y1,x2,y2) returns just that rectangle. Re-checking one field or one dialog costs a fraction of a full screenshot to send and to read — measured 4,244 bytes against 26,404 for the same moment's whole window.

• A batch of steps in one call: run_steps([...]) performs a list of actions in order with a pause between them, so a familiar app doesn't cost a round trip per click. The whole script is validated before any of it runs, every step is journaled with its own before/after frames, and check steps stop the run when the screen didn't do what the script predicted. Only the first step gets the stale-coordinate guard — see Known limitations.

• Waiting is never automatic: wait_stable() polls until the window stops repainting, but only when asked. Nothing sleeps behind the caller's back, so timing stays a visible decision rather than a hidden constant.

• Rolling journal: every tool call is appended to a throwaway session folder under %TEMP%\winauto-mcp\ with downscaled before/after JPEGs, kept for the last 5 sessions. history() lists the steps (failures included) and replay_frame(seq) returns the screen as it actually was at that step, so "what happened before this?" is answerable from evidence rather than recall.

• Reads work while the window is covered: capture goes through PrintWindow(PW_RENDERFULLCONTENT) first, which asks the window to render itself, and falls back to scraping the screen only if that returns nothing usable. Verified against Blender 5.2 (OpenGL), the Godot 4.6 editor and Windows 11 Notepad while each was fully covered by another window. Only input still needs the window in front.

• The pointer is shared: click/drag/scroll put the mouse back where the person left it (keep_cursor=true opts out, for modal tools that keep following the pointer).

• So is the foreground, and it is given back by itself: input needs the target window in front, so an action raises it — and hands the desktop straight back when the action ends, so a key typed in the gap between two actions lands where the person is looking instead of in the app being driven. run_steps hands back once at the end, not between steps. It refuses to hand back while a menu is open (a menu dies when its owner loses focus, and an open menu blocks foreground changes desktop-wide anyway) without forgetting what it owes, and forgets rather than yanks if the person has already moved somewhere else. Menus the app draws itself — XAML/WinUI, Electron, Qt, games — cannot be detected; keep_foreground(true) is the way to hold the window for an interaction that spans several calls (tests\diag_focus_return.py, 18 checks).

• Attaching is not taking over: attach_window only chooses which window the other tools mean. It does not raise the window and draws no outline — the window is raised, and the green outline appears, by itself at the first input, because that is the first moment input actually needs it. release_control() puts both back. This costs nothing: every input path already raised the window itself, so the raise was being paid twice. Measured — attach 6.0 ms lazy vs 26.7 ms eager, and the first click pays 11.9 ms more than a later one, so the change is ~9 ms faster overall and a read-only run never pays it at all (tests\diag_attach_cost.py).

• The outline only repaints when something moved: it used to redraw a transparent topmost window the size of the target on every 150 ms poll, for the whole session — enough compositor churn to make the tracked app stutter and the outline flicker, which reads as the automation having hung. Now the poll compares against what is already drawn and touches Tk only on a real change: measured 1 repaint over 3 idle seconds, against ~20 before, while a window that does move still updates (tests\diag_overlay_paint.py). It also cannot take the foreground: showing a Tk window activates it, so the outline was stealing focus from the very window it was outlining — found when a focus check reported a handle that was neither the person's window nor the app's, and printing its class named it. A decoration holding the foreground means keystrokes are aimed at a rectangle. The first fix for this was wrong and passed its diagnostic anyway for a day: it marked the handle Tk returns from wm_frame(), which is not the window that ends up on screen. What works is winfo_id() walked up with GA_ROOT, on the Tk thread, which resolves correctly even while the window is withdrawn — so the style goes on before the window is ever mapped and there is no first showing to race (tests\probe_overlay_activation.py asks this of the window actually visible, over three show/hide cycles).

• The keyboard is shared too, and the tool knows whose keystroke is whose: every event this server sends carries a signature Windows delivers untouched, so a key event can be attributed to the person, to us, or to a third injector. (Windows' own "injected" flag cannot do this — an on-screen keyboard, a remote-desktop session and another automation tool all set it.) While an action runs, the person's keys are held out so their typing cannot land in the middle of ours; ours pass the same block. Nothing is stored — not the key, not the character, only a count of "a human key happened" and when. A hook that kept key codes would be a keylogger, and the only honest way to promise it is not one is for the data never to exist.

• The keyboard always comes back, by five routes that do not depend on each other: the block is a lease that expires by itself within 20 seconds with no release call and no working server needed; three Escapes inside 1.5 s release it and latch it off until release_keyboard(); the mouse keeps working (only hover ever takes it, below); Windows discards a hook that is too slow and all hooks of a process that exits; and Ctrl+Alt+Del is beneath any hook by OS design. Nothing is installed at all until the first input is sent.

• hover is the one tool that pins the mouse, and only for its dwell: otherwise a hand on the mouse — or the person's own drift — slides the pointer off the target and the picture is of nothing. A low-level hook returning 1 genuinely pins the cursor rather than just hiding events from apps (measured); ClipCursor was rejected because that state belongs to no process, so a crash mid-hold would trap a stranger's pointer. The hold is a 3-second lease, is refused outright while a mouse button is physically down (that is a drag, and interrupting it strands it), is released by the same three Escapes, and reads no pointer coordinate at all. Every refusal still performs the hover and says the pointer was not pinned (tests\probe_mouse_lock.py, tests\diag_hover.py).

Related MCP server: helix-pilot

Tools

Setup

Full instructions — requirements, a client-free verification step, per-harness config and troubleshooting — are in SETUP.md. The short version:

From the repo root:

python -m venv .venv
.venv\Scripts\pip install -r requirements.txt

For what each tool takes and returns, and how to use them well, see HOWTOUSE.md.

Register with an MCP client

A harness launches the server from a working directory you do not control, so these configs need an absolute path. Print yours from the repo root and substitute it for <install-dir> below:

(Resolve-Path .).Path

Claude Desktop / Claude Code mcpServers config:

{
  "mcpServers": {
    "winauto": {
      "command": "<install-dir>\\.venv\\Scripts\\python.exe",
      "args": ["<install-dir>\\server.py"]
    }
  }
}

Claude Code CLI, run from the repo root — PowerShell expands these before claude sees them:

claude mcp add winauto -- "$((Resolve-Path .\.venv\Scripts\python.exe).Path)" "$((Resolve-Path .\server.py).Path)"

Codex Desktop %USERPROFILE%\.codex\config.toml:

[mcp_servers.winauto]
command = '<install-dir>\.venv\Scripts\python.exe'
args = ['<install-dir>\server.py']
enabled = true
startup_timeout_sec = 30

After saving the config, restart Codex Desktop or open a new task so the MCP server is loaded into the tool list. Verify the connection from a Codex task by searching for winauto tools or calling list_windows; a successful connection returns visible top-level windows with hwnd, title, and process fields.

If list_windows is not available, the active Codex task did not load the MCP server yet. Recheck the path to .venv\Scripts\python.exe, make sure server.py imports cleanly, then restart Codex again.

Known limitations (v0.1)

• One attached window at a time.

• SendInput is userspace input — most apps and browsers receive it fine, but some DirectInput/XInput-hooked fullscreen games may ignore it.

• SetForegroundWindow's foreground-lock rejection is mitigated (see AttachThreadInput note above) but not guaranteed in every edge case; attach_window/click/type_text don't hard-fail if it's still refused.

• UIA tree walk is capped (depth 15 / 3000 nodes / 150 results) to stay fast on deep trees (e.g. Chrome with heavy pages) — very deeply nested elements past the cap won't show up in the text summary.

• click_element/wait_for match by UIA Name (exact, then substring, case-insensitive); if multiple elements share a name the first one found in tree order is used — check get_elements if you need to disambiguate.

• Some apps with a custom-drawn title bar (e.g. Windows 11's modern Notepad) report menu items slightly outside GetClientRect, so their rect's y can be negative — a UIA quirk of that app, not a bug in this server.

• diff_since_snapshot can false-positive on apps that redraw part of their own UI continuously with no user input (e.g. a game/tool-script gizmo, blinking cursor, live counter) — confirmed against Godot's editor, whose 2D viewport keeps redrawing a debug hint label for the currently selected node. A nonzero changed_bbox means "something changed," not necessarily "your action caused it" — sanity-check that the bbox's location/size fits the expected effect.

• The location cache (remember_location/recall_location) is stored in .location_cache.json next to the server, keyed by process name + client size + label. It is never trusted blind: every recall_location call re-scans the live screen and compares against the cached bbox before returning a coordinate, so a stale entry (app updated, window resized, target scrolled out of view) is detected and dropped rather than causing a misclick.

• A change bounding box is one box around every changed pixel, so two small changes far apart produce a box covering all the untouched space between them, and its centre is very often a pixel that never changed. Typing one word into Windows 11 Notepad does this: the text moves at the top-left, the tab gains a modified marker, and the status bar's Ln/Col readout updates at the bottom-right — measured as three clusters totalling ~4,600 changed pixels inside a box of ~1.6 million. A large box means "the change is spread out", never "everything changed". Pass region to diff_since_snapshot and re-run over parts of the window to find where the change actually is. Applies equally to wait_stable's last_change_bbox.

• Background input is not possible for the apps this server targets. Posted window messages (WM_MOUSEMOVE/WM_LBUTTONDOWN/WM_CHAR, sent and posted, with and without a spoofed WM_ACTIVATE) were measured against Blender 5.2, the Godot 4.6 editor and Windows 11 Notepad: none moved a single pixel. Only Godot reacted at all, and only by brightening its title bar. So input still raises the window and borrows the real pointer — mitigated by restoring the cursor and by release_control(), not eliminated. True side-by-side use would need a separate Windows session or VM. (tests\spike_background*.py reproduce these measurements.)

• run_steps can only guard its first step. Steps 2..n act on a screen the script itself changed, which the caller has never been shown, so there is nothing to compare their coordinates against — they are a prediction of where the app will put things, which is exactly the assumption the stale-target guard otherwise refuses to make. This is a real loss of safety traded for the round trips, not an oversight: script only sequences already watched working, keep them short (the 40-step and 60s-of-waiting caps exist to make "short" the path of least resistance), and put a {"do":"check","region":[...],"expect":"changed"} after any step the rest depends on so a wrong prediction stops the run. Every step is journaled with before/after frames, so a script that goes wrong is reconstructable after the fact with history()/replay_frame() — that is the recovery path, not prevention.

• Blocking the keyboard means a machine-wide hook. Once the server sends its first input it installs a low-level keyboard hook, and such a hook sees every key event on the computer — not just those going to the target window. That is the mechanism; there is no window-scoped version of it. It stores nothing and swallows only the person's keys, only while an action is running, and it is gone when the process exits. But the exposure is real and worth stating rather than burying: while this server is running and has sent input, a bug in it is a bug in the path every keystroke on the machine takes. That is why the decision logic is kept trivial and is tested against synthetic events, why a thrown exception inside it passes the key through rather than eating it, and why release_keyboard(enable_blocking=false) exists to switch the whole thing off. Ctrl+Alt+Del cannot be blocked by it in any case.

• Looking at a region only counts for that region. After capture_region(toolbar), clicking in the toolbar goes through and clicking anywhere else in the window is refused with the list of rectangles actually looked at. Only the last 8 views are kept, and a new view supersedes any older one it fully contains — so a long run of small crops eventually forgets the earliest ones and a click there will ask to be re-checked.

• recall_location's re-verify margin can itself cause a false "stale" report if set too large for the target's surroundings: expanding into a neighboring label/icon merges them into one bigger bbox, which fails the size-ratio check even though the actual target never moved — confirmed against Blender's tightly packed viewport header (View Select Add Object, items only ~15-20px apart). Use a small margin (5-10px) for dense menu/toolbar rows; the default (15) assumes moderate spacing.

Agent usage notes for ZiiDMS desktop testing

When using this MCP from Codex/LLM sessions for ZiiDMS desktop UI testing, follow this sequence:

1. Verify the MCP is actually exposed in the active session before falling back to ad-hoc Python calls. Search or inspect available tools for a namespace like mcp__winauto. A local clone of this repo is not enough by itself; the client session must be restarted or configured so the MCP server is registered.

2. Start with list_windows. If the target app is missing, launch the app first, wait for the top-level window, then run list_windows again. Do not infer failure from Get-Process.MainWindowHandle = 0 alone; a visible attachable window may still appear in list_windows.

3. Attach by HWND with attach_window, then immediately call capture_screen. Use the full returned image and UIA summary as the test baseline. Do not rely on cropped screenshots from a chat transcript to decide whether buttons are missing.

4. Prefer click_element(name) for named buttons such as Save, Delete Selected, Restore Selected, New Department, and Include Deleted. For grid rows or custom-drawn cells, use coordinates from the full client-area image, or locate_in_region on a small candidate area; do not eyeball coordinates from resized display crops.

5. For ZiiDMS launched through DmsEnv, the normal launcher may start through a hidden wrapper. If list_windows does not show ZiiDMSNextGen Local, cleanly stop the prior client PID and relaunch through the approved launcher path. If a temporary visible-wrapper workaround is used, restore the launcher file immediately after launch.

6. Use capture_screen or get_elements after each action to verify command enabled/disabled state from UIA, especially for Save, Delete Selected, and Restore Selected. For visual-only state such as selected rows or deleted-row styling, use the full screenshot.

7. If a click unexpectedly opens a discard/confirmation prompt, record that as behavior evidence before dismissing it. In ZiiDMS Departments testing, read-only/protected fields should not mark the detail dirty merely because typing was attempted.

For the local ZiiDMS NextGen DMSEnv Departments screen, the common launch target is:

& 'C:\Users\Ping\.dmsenv\repos\eziidms-nextgen-dmsenv-a59a578e\run-client.ps1' -BypassLogin -ApiServer 'https://localhost:7160' -OrganizationId 7000 -DealershipId 7001 -Screen DepartmentsView

After launch, attach to the ZiiDMSNextGen Local window from list_windows and capture the full client area before testing.