uia-reader

An MCP server that reads a Windows window's real content via the UI Automation API instead of screenshotting it and asking a vision model to guess.

Why

A screenshot is the screen's raw pixel data — that's already "the binary." What's actually missing is the structured layer above it: the control tree Windows itself maintains for every window (control types, names, text content, state). Browsers already get this treatment (read_page reads the accessibility tree instead of screenshotting a tab); this does the same thing for native desktop windows.

Validated 2026-08 against a real CCD (Claude Code Desktop) window: the full chat transcript, message action buttons, live streaming status, token counts, and model/ effort settings were all readable as exact structured text — no vision, no guessing.

Related MCP server: uiautomation-mcp

Setup

pip install -r requirements.txt

Register as an MCP server

Register at user scope (-s user) so it's available from any project, not just the directory you happened to run the command from — claude mcp add's default scope is local-to-the-current-project, which is easy to get wrong by accident.

claude mcp add uia-reader -s user -- python "C:\path\to\uia-reader\server.py"

Replace C:\path\to\uia-reader with wherever you cloned this repo.

Tools

• list_windows(only_visible=True) — list on-screen windows with title + control class. Use this first to find the exact title to pass to the others.

• read_window(title, max_depth=40) — full structured content dump for a window whose title contains title (case-insensitive substring, anywhere in the title — not anchored to the start; see the title-matching note below). Meaningful controls only (empty layout wrappers are traversed but not printed), and text-bearing controls (Edit/Document/ComboBox) show their actual value inline, not just their label. Default depth is deliberately generous (40) — Chromium-based browser windows nest real content far deeper than native apps typically do.

• find_in_window(title, query, max_depth=40) — case-insensitive substring search across a window's content; faster than a full dump when you just need to know if/where something appears.

• click_element(title, name, control_type=None, exact_match=True, index=None) — invoke/click a control by its exact visible name. Ambiguous matches (more than one element with that name) click nothing and list candidates by index instead of guessing.

• type_into_element(title, name, text, control_type="Edit", exact_match=True, index=None) — write text into an editable control. Same disambiguation rule as click_element.

• ocr_window(title, region=None, save_image=True) — read the text in a window's pixels with Windows' built-in OCR, for windows that expose nothing to UI Automation (games, canvases). Read-only. Returns (x,y) text lines in window pixels and saves the captured image so a caller can check a value or look at icons. See below.

• click_window_point(title, x, y, method="message", show_cursor=True, wake=True) — click a point in a window that has no controls, in the same window-pixel frame as ocr_window. Sends exactly one click, then reports whether the screen changed. See below.

• press_key(title, key) — press one key in a window without focusing it (space, enter, esc, 0-9, a-z, arrows, f1-f12...). Games often print a key hint on each button.

• drag_window_point(title, x, y, dx, dy, steps=12) — drag with the real mouse, for lists and maps in the same kind of window. Same protections as a borrowed click, plus it lets go the moment the user moves the mouse. See below.

• learn_digits(title, region, text, font="default", ink=None) and read_digits(title, region, font="default", ink=None) — read numbers in a game's own pixel font by matching character shapes taught from numbers you know, instead of OCR. Anything unclear comes back as ?. See below.

Clicking and pressing keys in a window with no controls (2026-09-29)

Added so a session could play a Unity game without taking over the user's screen. Read the button's label with ocr_window first: in a game there is no undo, and a "?" button whose hint said "End" ended a run.

Unity ignores input until it's told it's active. Posted keys and clicks did nothing at first. Sending WM_NCACTIVATE, WM_ACTIVATEAPP, WM_ACTIVATE and WM_SETFOCUS first (as messages only; nothing comes to the front) made keys and ordinary buttons work.

Two click methods, because some controls read the real cursor. method="message" (the default) posts the click and never touches the mouse. It worked for ordinary buttons, but the game's scrolling-list items, skill-slot targets and upgrade-board tiles ignored it. method="mouse" borrows the real pointer for about 0.1 s, clicks, then puts the pointer and the user's foreground window back. It waits until the mouse has been still, with no button held, for 0.3 s, and refuses after 3 s: a borrow that landed during the user's own click scrolled the game's board away. Some buttons ignored both methods (a confirm dialog); its key hint (Space) worked, so prefer press_key whenever a button shows one.

A real click lands in whatever is on top, not in the window it was aimed at. So before borrowing the pointer, method="mouse" checks which window is on top at that spot. If anything else covers the target there (the Claude app, a browser with a bank open), or the target is minimized, it refuses and the pointer never moves. It checks before waiting for the mouse to be idle, and again just before clicking. Added 2026-09-30 in review, because the first version would have clicked into whatever covered the game. Tested on stand-in windows: a covered spot was refused, neither window was pressed, and the pointer-move function was called 0 times (the calls were recorded, not performed). The ghost marker, which is click-through, doesn't count as a cover. A real borrowed click on an uncovered spot wasn't re-run, because it would move the user's pointer.

A synthetic touch is not a separate pointer (tested 2026-09-30). Windows can create its own touch device (CreateSyntheticPointerDevice + InjectSyntheticPointerInput), which looked like a way to click without borrowing the mouse. On throwaway windows it pressed the target once, but Windows moved the real cursor to the touch point and left it there, and brought the touched window to the front. That's worse than method="mouse", which waits for the mouse to be idle and puts both the pointer and the foreground window back. Not built. Don't retry it expecting a second pointer.

Wake the window before a real click, too (2026-09-30). The activation messages above were only sent for message clicks. Watching a session play showed Unity also ignores a REAL mouse click while it thinks it's in the background: four clicks on a world-map icon in a row did nothing, then the session sent the activation messages first and the next click opened the page. So click_window_point now takes wake=True (the default) and sends them before either kind of click. Set it False only for a window that misbehaves when told it's active. What the stand-in tests can show is that the messages are sent (once with wake=True, never with False); they can't reproduce Unity's timing, so the evidence that it fixes the game is the session's own before-and-after.

Dragging (2026-09-30). drag_window_point replaces the drag the game session had written for itself, which moved the pointer with no cover check and no way to notice the user. It only uses the real mouse (the game's lists ignored message input and the wheel alike), waits for the mouse to be idle, refuses if another window covers the start, middle or end of the path, wakes the window, and puts the pointer and foreground window back. If the user moves the mouse mid-drag, it releases the button at once and says STOPPED PART-WAY, and it never retries: a partial drag has already moved something. Tested on a stand-in pad that records every mouse down, move and up: one down, one up, exact start and end points; an interrupted drag released at step 4 of 12 with exactly one press; a covered end point was refused with the pointer moved 0 times.

"Screen changed" is a hint, not proof. All three tools re-capture for 1.5 s and compare frames. A click can land with no visible change, and an animation can change the screen with no click, so read the screen again before retrying.

The change check was rebuilt (2026-09-30), because the game animates on its own. The old check averaged the difference over the whole window. Measured on the game sitting idle, the screen pulses by 25-45 levels over about 25,000 pixels roughly once a second, and that came close to the old threshold. A stand-in with a pulsing panel proved it: the old rule called a click on empty space a change. The new check:

• ignores anything that was already moving in a second capture taken 0.1 s before the action;

• counts only strong changes (a colour channel moved by more than 60 of 255);

• confirms on size (0.1% of the window, at least 300 pixels, or 150 near the click), or on a smaller change that holds still for three looks in a row (a counter's digit is ~110 pixels);

• says how many pixels changed and where, so the caller can tell whether it was the expected effect or something elsewhere. It leans towards "not confirmed", because a false "changed" would hide a missed click. On the idle game, 36 trials of 1.5 s with no action nearby: each rule fired once, on the same trial. The old rule could not do the stand-in's empty-space test without a false "changed"; the new one passes it (17/17 stand-in tests, including that one).

Message clicks reach the window, not the controls inside it. In a Windows Forms stand-in, a message click aimed at a button went to the form underneath; the button never saw it. That's fine for a Unity game, which is one window, but in an ordinary app use click_element or method="mouse" for a child control.

Tested live on the game: message clicks and keys on ordinary buttons, mouse clicks on list items, slot targets and board tiles, 15 mouse clicks in one session (1 ignored, the confirm dialog above), and the idle check with the mouse at rest.

Reading numbers in a game's own font (2026-09-30)

Windows OCR is built for ordinary text, and a game's pixel font defeats it exactly where it matters: slashed zeros, a % sign, digits packed tight. learn_digits / read_digits match character shapes instead. A game draws each digit identically every time, so shapes taught once from numbers you know (read them yourself from the saved image) recognise it from then on.

• Teach: learn_digits(title, [x0, y0, x1, y1], "2100", font="mygame"): a small margin around the number, nothing else inside. It refuses if the number of shapes it finds doesn't match the text, rather than mislabel them. Shapes are saved to fonts/<font>.json next to the server (per-machine data, not in git). Teach each character from two or three screens.

• Read: read_digits(title, region, font="mygame"): a character comes back only if its best match is strong (pixel overlap at least 0.7) and beats every other character by at least 0.05. Otherwise it's ?, with its closest candidate, and the result says not to act on the number as read. A font can set a stricter bar for itself: add "cutoff": 0.8 at the top level of its JSON file.

• Busy backgrounds: pass ink=[r, g, b], the number's colour, to ignore everything else. Thin slivers next to a number (the edge of an icon) are dropped, unless the sliver is plainly a stroke of the number: as tall as the digits, on their rows, and not cut by the region's edge. Art that lies mostly above or below the digits' rows is dropped too. A blob much wider than the widest normal character is two characters touching and is cut at its emptiest columns.

Tested on one game's header numbers, from saved screenshots (nothing sent to the game): taught from 6 screens, then tested on 11 others it had never seen, including red numbers beside a flame icon, a dimmed screen, and a + it was taught once:

The eight silent misreads are the reason for this tool: a plausible wrong number is worse than none. Then read live through the tool: 289, 28, 10350 and 179% (red, beside the flame), all exact.

Three silent misreads found in real use, and fixed (2026-10-07). A second number on the same game, small white counts drawn over artwork, read wrong in ways the header test never showed:

• A "1" drawn 2 pixels wide vanished. The sliver rule dropped it with no ?, so 41 read as 4 and 17 as 7. A thin run is now kept when it's as tall as the digits and on their rows.

• "19" came out as "1 9". The empty columns either side of a thin 1 were measured as a space. A caller taking the last group would have read 9. Gaps now treat a narrow character as a full-width cell.

• A taught sample was wrong. One "9" in the font was a 7, so 47 read as 49. Shape matching is only as good as what it was taught: read the true value from an enlarged crop, not from the tool. That font's 5 and 6 also overlap at 0.71, just over the default bar, which is why a font can now carry its own cutoff.

Held-out test on 47 reads from screenshots the font was never taught from:

The header test above is unchanged by these fixes (41 right, 3 flagged, 0 wrong).

What didn't work, and why the design is what it is. The first version shrank each character to 12x16 and averaged the difference over every pixel. It never misread either, but it couldn't tell an 8 from a slashed 0: the shared outline drowned out the few pixels that differ, and comparing at other sizes didn't help. Comparing at full size by overlap of inked pixels only (a 1-pixel shift allowed) is what fixed it. The splitter first judged "too wide" against the median width, which let the narrow 1s in "114" make its 4 look like two characters; it now compares against the widest normal character.

OCR for windows that expose nothing (2026-09-26)

Added for a Unity game: read_window returned only the title bar, because the game paints every button and number itself. ocr_window captures the window and runs Windows.Media.Ocr on it — local, no network, about a second for a whole window.

Capture. PrintWindow with PW_RENDERFULLCONTENT first, which reads the window even when something covers it (it worked for the Unity game). If that returns a black frame, it falls back to copying the screen and says so, since a screen copy includes whatever is on top. A minimized window is refused: it has no pixels.

Two passes, because one setting didn't read everything. Measured on the game's screens:

So the tool runs the plain pass, then threshold passes that only add lines where nothing was read already.

Known weak spot: slashed zeros. The same game's larger number panels use a zero with a slash through it. Every setting tried read it as 8, B or ø: 862000 came back as 8688B, 1260 as 126B. Digits in a font like that are unverified until checked against the image.

region, and why checking the image is cheap. Image cost to a vision model scales with area. The whole 1294x1039 game window is roughly 1,500+ tokens to look at; its 370x102 number panel is roughly 50, and read perfectly by eye. So the pattern that works is: OCR the window for labels and layout, then region=[x0,y0,x1,y1] on just the numbers and look at that small saved image.

Which window. title follows the same rule as every other tool; see "Which window a title means" below.

Tested: whole window, region, region too short, region outside the window, missing window, and a minimized stand-in window (a throwaway Windows Forms window, not anything real).

Real bugs found and fixed during testing (2026-08-24)

• Title matching must be substring, not prefix-anchored. An anchored ^title.* match breaks the instant a title gains a leading indicator — Notepad's unsaved-changes * was enough to make a previously-working title stop matching mid-session. Switched to case-insensitive substring matching everywhere.

• A control's accessible name often isn't its content. A Notepad text area's name is the static label "Text editor" — the actual typed text lives in the value, which read_window originally never looked at, so typed content was invisible to reads even though writes were working correctly. Fixed by pulling window_text() for value-bearing control types (Edit/Document/ComboBox) and showing it inline.

• Simulated keystrokes (type_keys) can silently corrupt text. Observed live: typing "...click/type test" landed as "...click/yype test" — a dropped/altered character with no error raised. type_into_element now tries atomic, pattern-based writes first (set_text(), then ValuePattern.SetValue() directly) and only falls back to simulated keystrokes — with an explicit warning in the response — if neither is supported by the target control.

Real limitation found via real use (2026-08-30)

Hit this trying to verify a background browser tab's state while driving the foreground tab with separate browser-automation tools: UI Automation reflects whichever tab is currently active/visible in a browser window — there is no way to target a specific background tab. Confirmed by direct reproduction: switching the active tab in an Edge window immediately changes what the window's title and UIA tree show, with zero indication of "this is stale" or "this is the wrong tab" — it just silently becomes the new tab's content. This is accurate to what's actually on screen (same as a screenshot would show), not a bug in the traversal logic, but it makes this server the wrong tool for reading browser tab content specifically — read_page/get_page_text (DevTools Protocol-based, not screen-state-based) are correct there since they don't depend on which tab is visually active.

All four tools now detect when the target window is a browser (Edge/Chrome/Firefox) and append an explicit warning to the response rather than returning ambiguous content silently — cheapest fix that doesn't require guessing at tab-targeting logic this server has no way to implement.

Rule: never use this tool to find, open, close or otherwise manage browser tabs. It has no concept of "whose tab this is" -- a query matches by name across the whole window, including tabs that belong to someone or something else, with nothing enforcing a boundary. A browser's own extension tooling that scopes actions to a single owner is the right place for tab management; this tool is for reading or clicking inside windows that aren't a browser at all, without ever touching the real mouse unless explicitly asked to.

Second real bug found via real use: max_depth was silently dead (2026-08-30)

Real-world use separately reported that reading a browser window with the (then-)default depth returned only toolbar/chrome, missing all real page content — and that raising max_depth to 40 fixed it. Checking the code before taking that fix at face value turned up something worse: max_depth was never actually passed into the traversal function at all — the parameter existed on read_window/find_in_window and did nothing, silently. The real cause of the missing content was almost certainly MAX_NODES (a flat 3000-node cap) being exhausted somewhere inside the browser's own toolbar/tab-strip subtree — which can easily contain thousands of nodes — before traversal (depth-first) ever reached the actual page content as a later sibling.

Fixed both halves: max_depth is now actually wired into the traversal and enforced (on structural/printed depth, matching what the indentation represents), and MAX_NODES was raised 3000 → 8000 to give real headroom. Verified live against a real Wikipedia article in a fresh Edge window: the full article body, references, and category list all came through correctly where the old code returned only browser chrome, with no truncation marker hit either way. Also added the same MAX_NODES safety cap (with no depth limit — a real match can legitimately be very deep) to _find_elements, used by click_element/type_into_element, which had no bound of any kind before this.

Your mouse stays yours

click_element activates controls without touching the real mouse pointer. It tries, in order, every UI Automation method that acts on a control directly:

1. invoke — buttons that support it

2. toggle — checkboxes and toggle buttons

3. select — tabs, radio buttons, list items

4. expand / collapse — menu buttons, combo boxes, tree nodes

5. default action — the control's own "press" through the accessibility layer

Where a control exposes state, the result is confirmed against that state: a toggle must actually flip, a menu must actually open, a tab must actually become selected. A method that reports success while the state doesn't change is treated as a failure and the next one is stopped at, not retried with another method — see At most one press below. Invoke and default action expose nothing to confirm against, and the response says so rather than claiming more.

The real pointer is used only with allow_mouse=True, and even then it is put back where the user left it — including when the click itself fails. Without that flag, a control with no cursor-free method is refused and nothing is clicked.

Before 2026-09-12 this tool tried invoke() and then went straight to the real mouse. The controls that took the user's cursor that day — Edge tabs, a desktop app's menu button — all supported a cursor-free method it simply never tried.

At most one press

A click is sent once. The tool moves on to the next activation method only when the current one could not be sent (the control doesn't support it). Once any method has been sent, it stops — even if the effect hasn't shown up yet — and says plainly whether the control's state confirmed it within a second.

Two ways a press can look unsent when it wasn't, and the tool stops on both:

• A slow effect. A menu button opens its menu a moment after Expand(). The first cursor-free version gave up on the state check after 80ms and tried the next method.

• An error after the call. An app can act on a press and then fail to answer. A stand-in control that counts presses and then throws was pressed twice by an earlier version — invoke, then its default action — which then reported "Nothing was clicked" (2026-09-14). Only a pattern the control doesn't offer at all counts as "not sent" now.

Why this is taken so seriously: on 2026-09-12 the calling session's own options menu, which contains Archive, was used as a test target, and the session was archived in the same second. The explanation first written down — a missed Expand() followed by a default action fired into the open menu — turned out to be impossible: the default-action step of that version could never run (see below). What pressed Archive is not established. The rules above close every double-press path found since.

A slow effect is not a missing effect, an error is not proof of nothing, and a second press is not a retry: it acts on whatever the screen shows now.

So an unconfirmed result tells you to check with read_window() rather than click again.

The default action never ran before 2026-09-14

The fifth step looked for iface_legacy_iaccessible, an attribute pywinauto 0.6.9 wrappers don't have. The error was caught like any unsupported pattern, so the step silently did nothing for every control, and the regression test "proving" the old double press gave its fake controls that attribute. It now goes through uia_defines.get_elem_interface(..., "LegacyIAccessible"). Edge's saved tab-group buttons need it: they offer expand/collapse plus a default action ("Press"), and when one reports itself as a leaf the expand step is skipped.

Tested against a Windows Forms stand-in whose controls count every press in the window title, read separately from the tool:

Where a session may act

Reading works in any window. Clicking and typing are scoped to the calling session's own window by default, with two separate opt-outs:

They are deliberately separate. Clicking or typing into another Claude session acts with that session's permissions rather than the caller's — a way around the permission decisions the user made for each session. The right move is almost always to send that session a message instead. allow_other_window alone does not unlock another Claude window.

A refused action clicks or types nothing, and says why.

Testing rule, learned the same afternoon: never exercise click tools against a real menu that contains destructive items (Archive, Delete, Send). Reproduce the behaviour with stand-in controls instead — they can record every press, which a real menu cannot.

The labelled marker

Every click and every typing action shows a small marker: an arrow with the calling session's name (WEB, OPS, …) in a stable colour per session. It starts in that session's own window, glides to the target, and fades — so you can watch which session is acting where without giving up your own cursor.

• Click-through and focus-free. It cannot be clicked and never becomes the active window; input passes straight through it.

• Truthful timing. The action waits for the glide to arrive, so the marker never shows something that already happened elsewhere.

• Multi-monitor. Positions use the same physical virtual-desktop coordinates UI Automation reports, so it lands correctly on offset and secondary monitors.

• Separate process. ghost_cursor.py runs detached with its output streams closed — this server speaks MCP over stdout, and a child inheriting that stream would corrupt every tool response.

Pass show_cursor=False to act without it, or caller="NAME" to override the label.

By default the label is the session's working-folder name. To use short names instead, copy callers.example.json to callers.json beside server.py and map folder names to labels (and optionally colours). callers.json is gitignored: it is a list of your own projects, which is private configuration, not code.

Which window a title means (2026-09-28)

Every tool finds its window by title. It used to take the first window whose title contained the text, in stacking order, with no check for a second match. So "MyGame" picked a chat window titled "MyGame research summary" over the game window titled exactly "MyGame" (found in real use, 2026-09-26). A read then silently describes the wrong window, and a click or type could act in it.

Now all six tools share one rule:

• an exact title wins (ignoring case, repeated spaces and invisible characters like the U+200B Edge hides in its titles);

• otherwise one partial match is used;

• several matches, or two windows with the same exact title, return a list and pick nothing. Pass more of the title.

Tested against throwaway Windows Forms windows, 15/15. Two of those checks were built to fail:

• the old code, given an ambiguous title, silently picked one;

• the new code, given the same title, refused, and click_element pressed nothing (the window's own counter stayed at 0).

A real window on the test machine also contained "research summary", and the tool listed both instead of picking one.

Known limitations

• Windows only (uses pywinauto's UIA backend). No macOS/Linux equivalent yet — would need the Accessibility API (AXUIElement) on macOS instead.

• Depends on the target app actually implementing accessibility support. Well-built Electron/Chromium apps (confirmed: CCD itself) expose a rich tree; a poorly-built or custom-canvas-rendered app may expose little beyond generic panes, in which case screenshots remain the fallback.

• An empty tree means "not exposed", not "empty" — and the two look identical. Custom-drawn/owner-drawn surfaces (canvases, diagram and arrangement views, some custom navigation lists) are painted as pixels with no UI Automation representation at all, so read_window succeeds and describes almost nothing, and find_in_window returns "no matches" for text that is plainly visible on screen. Reading that as a negative finding is the most expensive mistake this tool can cause, so since 2026-09-11 both tools say so explicitly when a visually substantial window exposes nothing beyond window chrome. One window can mix both kinds: standard controls elsewhere in it may still read perfectly. Read the window twice before concluding — a transient sparse read has been observed once.

• Control names are whitespace-normalised (newlines collapsed to spaces) for both display and matching, so the name a read tool prints is the name the click/type tools accept. Before 2026-09-11 these disagreed, and a wrapped label like "...keyboard and mouse\n(make this computer the server)" displayed one way and matched only another.

• No permission tiering. The built-in computer-use tool restricts what it'll click/ type per app category (e.g. browsers are click-blocked). This tool has no equivalent gating — if it's registered and callable, it can click or type into any window on the machine, including ones a more cautious policy would normally restrict. Be deliberate about when/whether this is registered for a session that also has broader autonomy.

• Click/type require an exact name match by default specifically because a wrong guess here has real side effects (unlike the read-only tools, which are safe by construction) — don't lower exact_match or blindly pick an index from an ambiguous list without confirming which element you actually want first.

License

MIT