# uia-reader

An MCP server that reads a Windows window's real content via the UI Automation API
instead of screenshotting it and asking a vision model to guess.

## Why

A screenshot is the screen's raw pixel data — that's already "the binary." What's
actually missing is the *structured* layer above it: the control tree Windows itself
maintains for every window (control types, names, text content, state). Browsers
already get this treatment (`read_page` reads the accessibility tree instead of
screenshotting a tab); this does the same thing for native desktop windows.

Validated 2026-08 against a real CCD (Claude Code Desktop) window: the full chat
transcript, message action buttons, live streaming status, token counts, and model/
effort settings were all readable as exact structured text — no vision, no guessing.

## Setup

```bash
pip install -r requirements.txt
```

## Register as an MCP server

Register at **user scope** (`-s user`) so it's available from any project, not just
the directory you happened to run the command from — `claude mcp add`'s default scope
is local-to-the-current-project, which is easy to get wrong by accident.

```bash
claude mcp add uia-reader -s user -- python "C:\path\to\uia-reader\server.py"
```

Replace `C:\path\to\uia-reader` with wherever you cloned this repo.

## Tools

- `list_windows(only_visible=True)` — list on-screen windows with title + control class.
  Use this first to find the exact title to pass to the others.
- `read_window(title, max_depth=40)` — full structured content dump for a window whose
  title contains `title` (case-insensitive substring, anywhere in the title — not
  anchored to the start; see the title-matching note below). Meaningful controls only
  (empty layout wrappers are traversed but not printed), and text-bearing controls
  (Edit/Document/ComboBox) show their actual value inline, not just their label. Default
  depth is deliberately generous (40) — Chromium-based browser windows nest real content
  far deeper than native apps typically do.
- `find_in_window(title, query, max_depth=40)` — case-insensitive substring search across
  a window's content; faster than a full dump when you just need to know if/where
  something appears.
- `click_element(title, name, control_type=None, exact_match=True, index=None)` —
  invoke/click a control by its exact visible name. Ambiguous matches (more than one
  element with that name) click nothing and list candidates by index instead of
  guessing.
- `type_into_element(title, name, text, control_type="Edit", exact_match=True, index=None)`
  — write text into an editable control. Same disambiguation rule as `click_element`.
- `ocr_window(title, region=None, save_image=True)` — read the **text in a window's
  pixels** with Windows' built-in OCR, for windows that expose nothing to UI Automation
  (games, canvases). Read-only. Returns `(x,y) text` lines in window pixels and saves the
  captured image so a caller can check a value or look at icons. See below.
- `click_window_point(title, x, y, method="message", show_cursor=True, wake=True)` — click a **point**
  in a window that has no controls, in the same window-pixel frame as `ocr_window`. Sends
  exactly one click, then reports whether the screen changed. See below.
- `press_key(title, key)` — press one key in a window without focusing it (space, enter,
  esc, 0-9, a-z, arrows, f1-f12...). Games often print a key hint on each button.
- `drag_window_point(title, x, y, dx, dy, steps=12)` — drag with the real mouse, for lists and
  maps in the same kind of window. Same protections as a borrowed click, plus it lets go the
  moment the user moves the mouse. See below.
- `learn_digits(title, region, text, font="default", ink=None)` and `read_digits(title, region, font="default", ink=None)`
  — read numbers in a game's own pixel font by matching character shapes taught from numbers
  you know, instead of OCR. Anything unclear comes back as `?`. See below.

## Clicking and pressing keys in a window with no controls (2026-09-29)

Added so a session could play a Unity game **without taking over the user's
screen**. Read the button's label with `ocr_window` first: in a game there is no undo, and a
"?" button whose hint said "End" ended a run.

**Unity ignores input until it's told it's active.** Posted keys and clicks did nothing at
first. Sending `WM_NCACTIVATE`, `WM_ACTIVATEAPP`, `WM_ACTIVATE` and `WM_SETFOCUS` first (as
messages only; nothing comes to the front) made keys and ordinary buttons work.

**Two click methods, because some controls read the real cursor.** `method="message"` (the
default) posts the click and never touches the mouse. It worked for ordinary buttons, but the
game's scrolling-list items, skill-slot targets and upgrade-board tiles ignored it.
`method="mouse"` borrows the real pointer for about 0.1 s, clicks, then puts the pointer and
the user's foreground window back. It waits until the mouse has been still, with no button
held, for 0.3 s, and refuses after 3 s: a borrow that landed during the user's own click
scrolled the game's board away. Some buttons ignored both methods (a confirm dialog); its key
hint (Space) worked, so prefer `press_key` whenever a button shows one.

**A real click lands in whatever is on top, not in the window it was aimed at.** So before
borrowing the pointer, `method="mouse"` checks which window is on top at that spot. If anything
else covers the target there (the Claude app, a browser with a bank open), or the target is
minimized, it refuses and the pointer never moves. It checks before waiting for the mouse to be
idle, and again just before clicking. Added 2026-09-30 in review, because the first version
would have clicked into whatever covered the game. Tested on stand-in windows: a covered spot
was refused, neither window was pressed, and the pointer-move function was called 0 times (the
calls were recorded, not performed). The ghost marker, which is click-through, doesn't count as a
cover. A real borrowed click on an uncovered spot wasn't re-run, because it would move the user's
pointer.

**A synthetic touch is not a separate pointer (tested 2026-09-30).** Windows can create its own
touch device (`CreateSyntheticPointerDevice` + `InjectSyntheticPointerInput`), which looked like a
way to click without borrowing the mouse. On throwaway windows it pressed the target once, but
**Windows moved the real cursor to the touch point and left it there**, and **brought the touched
window to the front**. That's worse than `method="mouse"`, which waits for the mouse to be idle and
puts both the pointer and the foreground window back. Not built. Don't retry it expecting a second
pointer.

**Wake the window before a real click, too (2026-09-30).** The activation messages above were
only sent for message clicks. Watching a session play showed Unity also ignores a REAL mouse
click while it thinks it's in the background: four clicks on a world-map icon in a row did
nothing, then the session sent the activation messages first and the next click opened the
page. So `click_window_point` now takes `wake=True` (the default) and sends them before either
kind of click. Set it False only for a window that misbehaves when told it's active. What the
stand-in tests can show is that the messages are sent (once with `wake=True`, never with
False); they can't reproduce Unity's timing, so the evidence that it fixes the game is the
session's own before-and-after.

**Dragging (2026-09-30).** `drag_window_point` replaces the drag the game session had written
for itself, which moved the pointer with no cover check and no way to notice the user. It only
uses the real mouse (the game's lists ignored message input and the wheel alike), waits for the
mouse to be idle, refuses if another window covers the start, middle or end of the path, wakes
the window, and puts the pointer and foreground window back. **If the user moves the mouse
mid-drag, it releases the button at once** and says `STOPPED PART-WAY`, and it never retries: a
partial drag has already moved something. Tested on a stand-in pad that records every mouse
down, move and up: one down, one up, exact start and end points; an interrupted drag released
at step 4 of 12 with exactly one press; a covered end point was refused with the pointer moved
0 times.

**"Screen changed" is a hint, not proof.** All three tools re-capture for 1.5 s and compare
frames. A click can land with no visible change, and an animation can change the screen with no
click, so read the screen again before retrying.

**The change check was rebuilt (2026-09-30), because the game animates on its own.** The old
check averaged the difference over the whole window. Measured on the game sitting idle,
the screen pulses by 25-45 levels over about 25,000 pixels roughly once a second, and that
came close to the old threshold. A stand-in with a pulsing panel proved it: the old rule called
a click on empty space a change. The new check:
- ignores anything that was already moving in a second capture taken 0.1 s before the action;
- counts only strong changes (a colour channel moved by more than 60 of 255);
- confirms on size (0.1% of the window, at least 300 pixels, or 150 near the click), or on a
  smaller change that holds still for three looks in a row (a counter's digit is ~110 pixels);
- says how many pixels changed and where, so the caller can tell whether it was the expected
  effect or something elsewhere.
It leans towards "not confirmed", because a false "changed" would hide a missed click. On the
idle game, 36 trials of 1.5 s with no action nearby: each rule fired once, on the same trial.
The old rule could not do the stand-in's empty-space test without a false "changed"; the new
one passes it (17/17 stand-in tests, including that one).

**Message clicks reach the window, not the controls inside it.** In a Windows Forms stand-in,
a message click aimed at a button went to the form underneath; the button never saw it. That's
fine for a Unity game, which is one window, but in an ordinary app use `click_element` or
`method="mouse"` for a child control.

Tested live on the game: message clicks and keys on ordinary buttons, mouse clicks on list
items, slot targets and board tiles, 15 mouse clicks in one session (1 ignored, the confirm
dialog above), and the idle check with the mouse at rest.

## Reading numbers in a game's own font (2026-09-30)

Windows OCR is built for ordinary text, and a game's pixel font defeats it exactly where it
matters: slashed zeros, a `%` sign, digits packed tight. `learn_digits` / `read_digits` match
character **shapes** instead. A game draws each digit identically every time, so shapes taught
once from numbers you know (read them yourself from the saved image) recognise it from then on.

- **Teach:** `learn_digits(title, [x0, y0, x1, y1], "2100", font="mygame")`: a small margin
  around the number, nothing else inside. It refuses if the number of shapes it finds doesn't
  match the text, rather than mislabel them. Shapes are saved to `fonts/<font>.json` next to the
  server (per-machine data, not in git). Teach each character from two or three screens.
- **Read:** `read_digits(title, region, font="mygame")`: a character comes back only if its
  best match is strong (pixel overlap at least 0.7) **and** beats every other character by at
  least 0.05. Otherwise it's `?`, with its closest candidate, and the result says not to act on
  the number as read. A font can set a stricter bar for itself: add `"cutoff": 0.8` at the top
  level of its JSON file.
- **Busy backgrounds:** pass `ink=[r, g, b]`, the number's colour, to ignore everything else.
  Thin slivers next to a number (the edge of an icon) are dropped, unless the sliver is plainly a
  stroke of the number: as tall as the digits, on their rows, and not cut by the region's edge.
  Art that lies mostly above or below the digits' rows is dropped too. A blob much wider than
  the widest normal character is two characters touching and is cut at its emptiest columns.

**Tested on one game's header numbers, from saved screenshots** (nothing sent to the game): taught
from 6 screens, then tested on 11 others it had never seen, including red numbers beside a flame
icon, a dimmed screen, and a `+` it was taught once:

| On the same 44 unseen numbers | `read_digits` | Windows OCR |
|---|---|---|
| Read exactly right | 41 | 18 |
| Read wrong, with no warning | **0** | **8** (e.g. 58500 as "58588", 50625 as "58625") |
| Flagged `?` / read nothing | 3 flagged | 18 empty |

The eight silent misreads are the reason for this tool: a plausible wrong number is worse than
none. Then read live through the tool: 289, 28, 10350 and 179% (red, beside the flame), all
exact.

**Three silent misreads found in real use, and fixed (2026-10-07).** A second number on the same
game, small white counts drawn over artwork, read wrong in ways the header test never showed:

- **A "1" drawn 2 pixels wide vanished.** The sliver rule dropped it with no `?`, so 41 read as 4
  and 17 as 7. A thin run is now kept when it's as tall as the digits and on their rows.
- **"19" came out as "1 9".** The empty columns either side of a thin 1 were measured as a space.
  A caller taking the last group would have read 9. Gaps now treat a narrow character as a
  full-width cell.
- **A taught sample was wrong.** One "9" in the font was a 7, so 47 read as 49. Shape matching
  is only as good as what it was taught: read the true value from an enlarged crop, not from the
  tool. That font's 5 and 6 also overlap at 0.71, just over the default bar, which is why a font
  can now carry its own `cutoff`.

Held-out test on 47 reads from screenshots the font was never taught from:

| | Right | Unread (`?`) | Wrong |
|---|---|---|---|
| Before | 24 | 17 | **6** |
| After | 45 | 2 | **0** |

The header test above is unchanged by these fixes (41 right, 3 flagged, 0 wrong).

**What didn't work, and why the design is what it is.** The first version shrank each character
to 12x16 and averaged the difference over every pixel. It never misread either, but it couldn't
tell an 8 from a slashed 0: the shared outline drowned out the few pixels that differ, and
comparing at other sizes didn't help. Comparing at full size by overlap of inked pixels only (a
1-pixel shift allowed) is what fixed it. The splitter first judged "too wide" against the
median width, which let the narrow 1s in "114" make its 4 look like two characters; it now
compares against the widest normal character.

## OCR for windows that expose nothing (2026-09-26)

Added for a Unity game: `read_window` returned only the title bar, because
the game paints every button and number itself. `ocr_window` captures the window and runs
`Windows.Media.Ocr` on it — local, no network, about a second for a whole window.

**Capture.** `PrintWindow` with `PW_RENDERFULLCONTENT` first, which reads the window even
when something covers it (it worked for the Unity game). If that returns a black frame, it
falls back to copying the screen and says so, since a screen copy includes whatever is on
top. A minimized window is refused: it has no pixels.

**Two passes, because one setting didn't read everything.** Measured on the game's screens:

| Pass | Labels ("Return to Title", "Save Data") | Thin pixel digits ("7900", "5.27%") |
|---|---|---|
| Plain, 2x upscale | read | **not read at all**, at any scale tried |
| Black/white threshold + thickened strokes | mostly lost | read exactly |

So the tool runs the plain pass, then threshold passes that only *add* lines where nothing
was read already.

**Known weak spot: slashed zeros.** The same game's larger number panels use a zero with a
slash through it. Every setting tried read it as `8`, `B` or `ø`: `862000` came back as
`8688B`, `1260` as `126B`. Digits in a font like that are unverified until checked against
the image.

**`region`, and why checking the image is cheap.** Image cost to a vision model scales with
area. The whole 1294x1039 game window is roughly 1,500+ tokens to look at; its 370x102
number panel is roughly 50, and read perfectly by eye. So the pattern that works is: OCR the
window for labels and layout, then `region=[x0,y0,x1,y1]` on just the numbers and look at
that small saved image.

**Which window.** `title` follows the same rule as every other tool; see "Which window a title means" below.

Tested: whole window, region, region too short, region outside the window, missing window,
and a minimized stand-in window (a throwaway Windows Forms window, not anything real).

## Real bugs found and fixed during testing (2026-08-24)

- **Title matching must be substring, not prefix-anchored.** An anchored `^title.*`
  match breaks the instant a title gains a leading indicator — Notepad's unsaved-changes
  `*` was enough to make a previously-working title stop matching mid-session. Switched
  to case-insensitive substring matching everywhere.
- **A control's accessible *name* often isn't its *content*.** A Notepad text area's
  name is the static label "Text editor" — the actual typed text lives in the value,
  which `read_window` originally never looked at, so typed content was invisible to
  reads even though writes were working correctly. Fixed by pulling `window_text()` for
  value-bearing control types (Edit/Document/ComboBox) and showing it inline.
- **Simulated keystrokes (`type_keys`) can silently corrupt text.** Observed live:
  typing "...click/type test" landed as "...click/yype test" — a dropped/altered
  character with no error raised. `type_into_element` now tries atomic, pattern-based
  writes first (`set_text()`, then `ValuePattern.SetValue()` directly) and only falls
  back to simulated keystrokes — with an explicit warning in the response — if neither
  is supported by the target control.

## Real limitation found via real use (2026-08-30)

Hit this trying to verify a background browser tab's state while driving the
foreground tab with separate browser-automation tools: **UI Automation reflects whichever tab
is currently active/visible in a browser window — there is no way to target a specific
background tab.** Confirmed by direct reproduction: switching the active tab in an Edge
window immediately changes what the window's title and UIA tree show, with zero
indication of "this is stale" or "this is the wrong tab" — it just silently becomes the
new tab's content. This is accurate to what's actually on screen (same as a screenshot
would show), not a bug in the traversal logic, but it makes this server the wrong tool
for reading browser tab content specifically — read_page/get_page_text (DevTools
Protocol-based, not screen-state-based) are correct there since they don't depend on
which tab is visually active.

All four tools now detect when the target window is a browser (Edge/Chrome/Firefox) and
append an explicit warning to the response rather than returning ambiguous content
silently — cheapest fix that doesn't require guessing at tab-targeting logic this
server has no way to implement.

**Rule: never use this tool to find, open, close or otherwise manage browser tabs.** It has no
concept of "whose tab this is" -- a query matches by name across the whole window, including tabs
that belong to someone or something else, with nothing enforcing a boundary. A browser's own
extension tooling that scopes actions to a single owner is the right place for tab management;
this tool is for reading or clicking *inside* windows that aren't a browser at all, without ever
touching the real mouse unless explicitly asked to.

## Second real bug found via real use: `max_depth` was silently dead (2026-08-30)

Real-world use separately reported that reading a browser window with the (then-)default
depth returned only toolbar/chrome, missing all real page content — and that raising
`max_depth` to 40 fixed it. Checking the code before taking that fix at face value
turned up something worse: **`max_depth` was never actually passed into the traversal
function at all** — the parameter existed on `read_window`/`find_in_window` and did
nothing, silently. The real cause of the missing content was almost certainly
`MAX_NODES` (a flat 3000-node cap) being exhausted somewhere inside the browser's own
toolbar/tab-strip subtree — which can easily contain thousands of nodes — before
traversal (depth-first) ever reached the actual page content as a later sibling.

Fixed both halves: `max_depth` is now actually wired into the traversal and enforced
(on structural/printed depth, matching what the indentation represents), and `MAX_NODES`
was raised 3000 → 8000 to give real headroom. Verified live against a real Wikipedia
article in a fresh Edge window: the full article body, references, and category list
all came through correctly where the old code returned only browser chrome, with no
truncation marker hit either way. Also added the same `MAX_NODES` safety cap (with no
depth limit — a real match can legitimately be very deep) to `_find_elements`, used by
`click_element`/`type_into_element`, which had no bound of any kind before this.

## Your mouse stays yours

`click_element` activates controls **without touching the real mouse pointer.** It tries, in
order, every UI Automation method that acts on a control directly:

1. **invoke** — buttons that support it
2. **toggle** — checkboxes and toggle buttons
3. **select** — tabs, radio buttons, list items
4. **expand / collapse** — menu buttons, combo boxes, tree nodes
5. **default action** — the control's own "press" through the accessibility layer

Where a control exposes state, the result is **confirmed against that state**: a toggle must
actually flip, a menu must actually open, a tab must actually become selected. A method that
reports success while the state doesn't change is treated as a failure and the next one is
stopped at, not retried with another method — see *At most one press* below. Invoke and
default action expose nothing to confirm against, and the response says so rather than
claiming more.

The real pointer is used **only** with `allow_mouse=True`, and even then it is put back where
the user left it — including when the click itself fails. Without that flag, a control with
no cursor-free method is refused and nothing is clicked.

Before 2026-09-12 this tool tried `invoke()` and then went straight to the real mouse. The
controls that took the user's cursor that day — Edge tabs, a desktop app's menu button — all
supported a cursor-free method it simply never tried.

### At most one press

A click is sent **once**. The tool moves on to the next activation method only when the
current one **could not be sent** (the control doesn't support it). Once any method has been
sent, it stops — even if the effect hasn't shown up yet — and says plainly whether the
control's state confirmed it within a second.

Two ways a press can look unsent when it wasn't, and the tool stops on both:

- **A slow effect.** A menu button opens its menu a moment *after* `Expand()`. The first
  cursor-free version gave up on the state check after 80ms and tried the next method.
- **An error after the call.** An app can act on a press and then fail to answer. A stand-in
  control that counts presses and then throws was pressed **twice** by an earlier version —
  invoke, then its default action — which then reported "Nothing was clicked" (2026-09-14).
  Only a pattern the control doesn't offer at all counts as "not sent" now.

Why this is taken so seriously: on 2026-09-12 the calling session's own options menu, which
contains **Archive**, was used as a test target, and the session was archived in the same
second. The explanation first written down — a missed `Expand()` followed by a default action
fired into the open menu — **turned out to be impossible**: the default-action step of that
version could never run (see below). What pressed Archive is not established. The rules above
close every double-press path found since.

A slow effect is not a missing effect, an error is not proof of nothing, and a second press is
not a retry: it acts on whatever the screen shows *now*.

So an unconfirmed result tells you to check with `read_window()` rather than click again.

### The default action never ran before 2026-09-14

The fifth step looked for `iface_legacy_iaccessible`, an attribute pywinauto 0.6.9 wrappers
don't have. The error was caught like any unsupported pattern, so the step silently did nothing
for every control, and the regression test "proving" the old double press gave its fake controls
that attribute. It now goes through `uia_defines.get_elem_interface(..., "LegacyIAccessible")`.
Edge's saved tab-group buttons need it: they offer expand/collapse plus a default action
("Press"), and when one reports itself as a leaf the expand step is skipped.

Tested against a Windows Forms stand-in whose controls count every press in the window title,
read separately from the tool:

| Case | Result |
|---|---|
| Old code, control with only a default action | not reachable, 0 presses |
| New code, same control | 1 press |
| Slow control (effect lands 1.5s later) | 1 press, none added afterwards |
| Empty default action | nothing sent |
| Normal control via `click_element` | invoke first, 1 press |
| **Must fail:** control that counts then throws, old code | pressed, reported as nothing sent |
| Same control, new code | **1 press**, reported as possibly taken effect |

### Where a session may act

Reading works in any window. **Clicking and typing are scoped to the calling session's own
window by default**, with two separate opt-outs:

| flag | allows |
|---|---|
| `allow_other_window=True` | an ordinary app window — a browser, a settings dialog |
| `allow_other_session=True` | **another Claude session's window** |

They are deliberately separate. Clicking or typing into another Claude session acts with
*that* session's permissions rather than the caller's — a way around the permission decisions
the user made for each session. The right move is almost always to send that session a
message instead. `allow_other_window` alone does **not** unlock another Claude window.

A refused action clicks or types nothing, and says why.

**Testing rule, learned the same afternoon:** never exercise click tools against a real menu
that contains destructive items (Archive, Delete, Send). Reproduce the behaviour with stand-in
controls instead — they can record every press, which a real menu cannot.

### The labelled marker

Every click and every typing action shows a small marker: an arrow with the calling session's
name (`WEB`, `OPS`, …) in a stable colour per session. It **starts in that session's own
window, glides to the target, and fades** — so you can watch *which* session is acting
*where* without giving up your own cursor.

- **Click-through and focus-free.** It cannot be clicked and never becomes the active window;
  input passes straight through it.
- **Truthful timing.** The action waits for the glide to arrive, so the marker never shows
  something that already happened elsewhere.
- **Multi-monitor.** Positions use the same physical virtual-desktop coordinates UI Automation
  reports, so it lands correctly on offset and secondary monitors.
- **Separate process.** `ghost_cursor.py` runs detached with its output streams closed — this
  server speaks MCP over stdout, and a child inheriting that stream would corrupt every tool
  response.

Pass `show_cursor=False` to act without it, or `caller="NAME"` to override the label.

By default the label is the session's working-folder name. To use short names instead, copy
`callers.example.json` to `callers.json` beside `server.py` and map folder names to labels
(and optionally colours). `callers.json` is gitignored: it is a list of your own projects,
which is private configuration, not code.

## Which window a title means (2026-09-28)

Every tool finds its window by `title`. It used to take the **first** window whose title
*contained* the text, in stacking order, with no check for a second match. So `"MyGame"`
picked a chat window titled "MyGame research summary" over the game window titled
exactly "MyGame" (found in real use, 2026-09-26). A read then
silently describes the wrong window, and a click or type could act in it.

Now all six tools share one rule:
- **an exact title wins** (ignoring case, repeated spaces and invisible characters like the
  U+200B Edge hides in its titles);
- otherwise **one** partial match is used;
- **several matches, or two windows with the same exact title, return a list and pick nothing.**
  Pass more of the title.

Tested against throwaway Windows Forms windows, 15/15. Two of those checks were built to fail:
- the old code, given an ambiguous title, silently picked one;
- the new code, given the same title, refused, and `click_element` pressed nothing (the window's
  own counter stayed at 0).

A real window on the test machine also contained "research summary", and the tool listed both
instead of picking one.

## Known limitations

- Windows only (uses `pywinauto`'s UIA backend). No macOS/Linux equivalent yet — would
  need the Accessibility API (`AXUIElement`) on macOS instead.
- Depends on the target app actually implementing accessibility support. Well-built
  Electron/Chromium apps (confirmed: CCD itself) expose a rich tree; a poorly-built or
  custom-canvas-rendered app may expose little beyond generic panes, in which case
  screenshots remain the fallback.
- **An empty tree means "not exposed", not "empty" — and the two look identical.**
  Custom-drawn/owner-drawn surfaces (canvases, diagram and arrangement views, some custom
  navigation lists) are painted as pixels with no UI Automation representation at all, so
  `read_window` succeeds and describes almost nothing, and `find_in_window` returns "no
  matches" for text that is plainly visible on screen. Reading that as a negative finding
  is the most expensive mistake this tool can cause, so since 2026-09-11 both tools say so
  explicitly when a visually substantial window exposes nothing beyond window chrome. One
  window can mix both kinds: standard controls elsewhere in it may still read perfectly.
  Read the window twice before concluding — a transient sparse read has been observed once.
- Control names are whitespace-normalised (newlines collapsed to spaces) for both display
  and matching, so the name a read tool prints is the name the click/type tools accept.
  Before 2026-09-11 these disagreed, and a wrapped label like `"...keyboard and
  mouse\n(make this computer the server)"` displayed one way and matched only another.
- **No permission tiering.** The built-in computer-use tool restricts what it'll click/
  type per app category (e.g. browsers are click-blocked). This tool has no equivalent
  gating — if it's registered and callable, it can click or type into *any* window on
  the machine, including ones a more cautious policy would normally restrict. Be
  deliberate about when/whether this is registered for a session that also has broader
  autonomy.
- Click/type require an exact name match by default specifically because a wrong guess
  here has real side effects (unlike the read-only tools, which are safe by
  construction) — don't lower `exact_match` or blindly pick an `index` from an ambiguous
  list without confirming which element you actually want first.

## License

[MIT](LICENSE)