# tandem-agent-mouse

> [!WARNING]
> # 🧪 BETA — v1.0.0b1
> **This is pre-release software.** Field-tested on Windows 11 26200
> (Bilibili, Word, Edge guest, four-round ZCode verification + a
> dual-round regression), but the API surface may still shift before 1.0.
> Pin your version and read the CHANGELOG before upgrading.

> **Background-first desktop automation for Windows as an MCP server: the
> agent sees and drives your apps — without ever disturbing you.**
>
> Lets Claude (or any MCP client) read any window's UIA accessibility tree,
> take background screenshots, and perform clicks and typing. A mode works
> on the user's desktop in the background — it never moves the real cursor
> or steals focus, and the forbidden APIs **raise on fetch**. B mode runs
> the agent's apps on a hidden second desktop that the user watches through
> a corner projection window; input is injected as window messages and the
> sandbox's whole process tree is guaranteed orphan-free by a Job Object.

**English** · [简体中文](README_zh.md)

**Version:** 1.0.0b1 **(beta)** | **Platform:** Windows 10/11 | **Tests:** 32

---

## Red lines (enforced in code, not by convention)

> Every red line below is structurally enforced by
> [guard.py](src/tandem_agent_mouse/win32/guard.py) — it is not a prompt-level
> promise:
>
> - **A mode never touches the user's desktop state.** `SetForegroundWindow`,
>   `SetCursorPos`, `SendInput`, `keybd_event`, `mouse_event`,
>   `AttachThreadInput`, `SetFocus` raise `ForegroundViolation` on fetch in
>   the parent process — not a runtime warning, a missing function pointer.
> - **The input desktop is NEVER switched — at any time.** `SwitchDesktop`
>   is blocked by the same guard. Making the sandbox the input desktop would
>   let a "real cursor" work there, but it locks the user out of their own
>   machine while switched. Sandbox "human-like" input is instead paced
>   message injection plus a **virtual cursor** drawn inside the projection
>   window — pixels in a 480 px corner window, never the user's pointer.
> - **Total background failure returns `needs_foreground`.** The user
>   decides whether to switch to B mode; the server never switches on its
>   own.
> - **B mode input is window-message injection only** (PostMessage into the
>   sandbox's windows). Measured on Win11 26200: SendInput always injects
>   into the INPUT desktop (Default while the user works) — from a sandbox
>   thread it fails with ACCESS_DENIED, and from the default desktop it
>   would move the user's real cursor. Message injection makes "never
>   touches the user's cursor" structurally true.
> - **Zero orphaned sandbox processes.** Every sandbox process starts
>   CREATE_SUSPENDED, joins a `KILL_ON_JOB_CLOSE` Job Object, then resumes.
>   Closing the projection window, `set_mode A`, graceful server exit, a
>   hard server kill — all four paths terminate the whole tree
>   (kernel-enforced).

---

## Why tandem-agent-mouse?

Existing desktop-automation approaches force a choice: pure vision
(screenshot + coordinate clicks — slow and brittle) or foreground takeover
(steals the mouse and focus — the user must stop working). This repository
provides a third path:

**The recognition layer is UIA, not pixels.** One unified element format
(shared by the UIA and visual paths):

```json
{
  "id": 17, "type": "button", "name": "OK", "value": null,
  "state": ["enabled", "offscreen"],
  "rect": [120, 80, 64, 24],
  "actions": ["invoke", "click", "toggle"],
  "depth": 3, "source": "uia"
}
```

Priority chain: **UIA CacheRequest full-tree scan → (empty tree) wake
probe + retry → (still failing) PrintWindow(PW_RENDERFULLCONTENT) →
(blank) Windows Graphics Capture → honest errors**. Screenshots return MCP
ImageContent plus grid metadata; the client reads the image, targets a
point, and sends it back through `act`'s `click_at`.

**A/B modes, one tool surface.** A mode is the default: background UIA
reads, background captures, batch actions that return only the diff. B
mode is opt-in: hidden sandbox desktop + corner projection — the user and
the agent each operate their own world, neither fighting for the mouse,
both able to watch (and nudge) the other.

---

## How it works

```mermaid
flowchart LR
    subgraph CLIENT["MCP client (Claude, ...)"]
        T["list_windows / observe / act /<br/>screenshot / set_mode /<br/>sandbox_*"]
    end
    T -->|stdio JSON-RPC<br/>zero SDK| SRV["server.py"]

    subgraph GUARD["win32/guard.py — red-line guard"]
        G["SetForegroundWindow / SendInput /<br/>SetCursorPos / keybd_event<br/>raise on fetch"]
    end

    SRV -->|"A mode"| UIA["uia/ persistent STA thread<br/>CacheRequest single-pass scan<br/>RuntimeId-stable identity"]
    UIA --> ACT["core/executor batch run<br/>invoke/toggle/set_value<br/>+ diff"]
    ACT -->|honest failure| NF["needs_foreground"]

    SRV -->|"B mode"| SB["sandbox/"]
    SB --> D["desktop.py<br/>CreateDesktopW hidden desktop<br/>+ Job Object membership"]
    SB --> IN["input_thread.py<br/>PostMessage injection<br/>(no cursor touched)"]
    SB --> PJ["projection.py<br/>topmost corner window<br/>frame pump 0.5s/frame"]
    PJ -->|"user clicks projection"| IN2["mapped back to sandbox<br/>coords, forwarded"]
    PJ -->|"user clicks X"| STOP["stop()<br/>TerminateJobObject<br/>whole tree dies"]
```

Three things the diagram makes explicit:

- **The guard is the bottom layer**: any code reaching for a forbidden API
  passes through guard.py first.
- **A-mode failures are honest**: a UIA pattern that will not resolve, a
  message click that Chromium refuses — reported as `needs_foreground`,
  never silently degraded into foreground theft.
- **B-mode shutdown is kernel-guaranteed**: the moment the projection
  window closes, the Job Object closes with it and every sandbox process —
  including the real process behind an alias stub — dies.

---

## Installation

```bash
cd "Tandem Agent Mouse"
uv pip install --system -e .
# Base (PrintWindow captures); add WGC for the Electron blank-frame fallback:
uv pip install --system winsdk
# Tests
python -m pytest tests/ -q
# stdio MCP server
python -m tandem_agent_mouse serve
```

### Registering in Claude Code / Claude Desktop

```json
{
  "mcpServers": {
    "tandem-agent-mouse": {
      "command": "python",
      "args": ["-m", "tandem_agent_mouse", "serve"]
    }
  }
}
```

Other MCP clients (ZCode and friends) take the same configuration.

---

## Quick start

Talk to the client in natural language. Typical flows:

**A mode (background, undisturbed):**

```text
"Write something in Notepad"  -> observe finds the edit area -> act set_value
"Show me this window"         -> screenshot returns image + grid + click_at usage
```

**B mode (sandbox + projection, you watch it work):**

```text
sandbox_mode "sandbox" -> the "Tandem Agent Mouse Sandbox" corner window appears
sandbox_launch Edge    -> Edge opens on the hidden desktop, visible in the projection
sandbox_click/type     -> message-injected search, navigation, playback
sandbox_human          -> paced, human-like input; virtual cursor drawn in the window
close the projection   -> the sandbox tears down, zero processes left
```

Live-verified example: Edge on the sandbox -> bilibili -> search
"Luo Xiaohei" -> open the Luo Xiaohei Zhanji playback page — the user's
cursor never moved and no focus was stolen at any point.

---

## MCP tools

Two parallel tool families — no A/B letters to remember:

- **`background_*`** — work on the USER's desktop. Never touches the cursor
  or focus; forbidden APIs raise on fetch.
- **`sandbox_*`** — work in the hidden sandbox desktop. The projection
  window shows it; all input is message-injected.

| Tool | Family | Description |
| --- | --- | --- |
| `background_windows` | background | Enumerate visible top-level windows (hwnd/pid/title/class/minimized) |
| `background_observe` | background | Structured element table for a window (UIA) |
| `background_act` | background | Batch action steps + incremental diff (invoke/click/toggle/set_value/click_at...) |
| `background_screenshot` | background | Background capture (PrintWindow→WGC); image + grid |
| `sandbox_mode` | sandbox | Start (`mode: "sandbox"`) or stop (`mode: "background"`) the sandbox + projection — user decision |
| `sandbox_launch` | sandbox | Launch a program on the hidden desktop (Chromium auto-wrapped: `cmd.exe /c` + `--force-renderer-accessibility`) |
| `sandbox_observe` | sandbox | UIA element table of a sandbox window |
| `sandbox_click` / `sandbox_type` | sandbox | Message-injected click / typing (no cursor, no switch) |
| `sandbox_scroll` | sandbox | Message-injected wheel scroll at a sandbox point (up/down, 1-10 notches) |
| `sandbox_wait` | sandbox | Poll the UIA tree until a condition holds (`text_exists`/`text_gone`) — the semantic cure for timing races; honest timeout, not an error |
| `sandbox_human` | sandbox | **Human-like pacing**: the same message injection with human-speed paths — every step is drawn as a virtual cursor (trail + click ripples) inside the projection window; no desktop switch, no real cursor, IME bypassed |
| `sandbox_screenshot` | sandbox | Capture the sandbox's current window; image + grid |
| `bridge_status` / `bridge_call` | legacy | windows-mcp bridge state / deprecated proxy call |

Legacy names (`list_windows`, `observe`, `act`, `screenshot`, `set_mode`)
remain as routing aliases for old clients but are no longer advertised.

`act` executes N steps per round trip and returns only the
added/removed/changed elements; the settle interval between steps is
configurable (default 350 ms).

---

## B mode v2: sandbox + corner projection

1. **Hidden sandbox desktop**: `CreateDesktopW` creates a second desktop;
   the agent's programs (e.g. Edge) open there via `STARTUPINFO.lpDesktop`,
   invisible to the user's desktop.
2. **Corner projection window**: a topmost bottom-right window (title
   "Tandem Agent Mouse Sandbox — 关闭=结束沙箱") fed by a frame pump thread
   that PrintWindows the sandbox's main window and blits it; user clicks
   inside the projection map back to real sandbox coordinates and are
   forwarded. A **virtual-cursor overlay** draws the agent's input steps
   (trail, cursor ring, click ripples) on top of the mirror, plus a status
   bar. **Closing the projection is the user's explicit decision to end the
   sandbox**; the pump detects the close and tears everything down.
3. **Message-injected input**: `sandbox_click/type` PostMessage
   WM_MOUSEMOVE/LBUTTONDOWN/CHAR directly to sandbox windows. No cursor
   moves, no focus changes — the user keeps working while the agent works
   "next door".

### Lifecycle (no orphaned sandbox processes, by construction)

- **Job Object membership**: `sandbox_launch` starts the process
  CREATE_SUSPENDED, assigns it to a `KILL_ON_JOB_CLOSE` job, then resumes
  it — the process cannot spawn children before it joins the job, so the
  whole tree is inside the net.
- **Four shutdown paths, all killing every sandbox process**:
  1. user closes the projection window → pump detects → `stop()` →
     `TerminateJobObject`;
  2. the agent calls `set_mode A` → `stop()`;
  3. graceful server exit (stdio EOF) → `atexit.register(_sandbox.stop)`;
  4. hard server kill (TerminateProcess/crash) → atexit never runs, but
     the dying process closes the job handle and
     `KILL_ON_JOB_CLOSE` fires in the kernel.
- Verified on a live machine: after `stop()`, after projection close, and
  after `taskkill /F` on the server — zero sandbox processes remain.

### Chelate Shell integration

This repository also ships as the `tandem-agent-mouse` skill of
[Chelate Shell](../Chelate_Shell/chelate-shell-v1.3.0/README.md)
(stage=`infrastructure`): `skills/tandem-agent-mouse/run.py` is a thin
adapter with zero copied logic — everything routes into this package's
modules; missing `comtypes`/`pillow` degrade to an `UNVERIFIED` gate
instead of crashing the pipeline.

---

## Field-tested pitfalls (all verified on real machines; do not re-learn them)

### Recognition layer (Windows 11 + Python 3.12)

1. **CacheRequest.TreeScope must include the element itself** (`7` =
   element+children+descendants). Passing `4` (subtree only) fails with
   `E_INVALIDARG`.
2. **Pattern availability flags** have no `CachedIs*PatternAvailable`
   property; read `GetCachedPropertyValue(30031..30044)`.
3. **Do not put Value(30012) in the CacheRequest**: Chromium pane elements
   reject the property, truncating the whole cached walk to one level.
   Read Value per element with `GetCachedPropertyValue`.
4. **All UIA COM calls run on one persistent STA thread** (com_worker).
   Per-call threads + CoInitialize/CoUninitialize make Chromium providers
   return NULL pattern pointers on the second call.
5. **IsEnabled=30010, BoundingRectangle=30001** — 30006 is AcceleratorKey;
   a wrong property ID only explodes when that element is read.
6. Chromium's accessibility tree is lazily activated: wake it with
   `ElementFromHandle` (sends WM_GETOBJECT), wait ~0.5 s, rescan; a
   minimized/restored app may sleep again.
7. Chromium background clicks: the browser chrome (title-bar buttons)
   accepts `WM_LBUTTON*`; web content does not (the renderer process owns
   input) — web content goes through UIA Invoke, and failure reports
   `needs_foreground` honestly.
8. `PrintWindow(PW_RENDERFULLCONTENT)` captures most Electron/Chromium
   windows; the occasional all-blank result falls through to WGC
   (optional `winsdk` dependency).
9. Cross-round element identity uses the UIA RuntimeId (stable across
   scans on Chromium in practice).

### Sandbox (Windows 11 26200)

1. **`CreateDesktopW` lives in user32**: old documentation says kernel32;
   on Win11 it is user32.
2. **`SetThreadDesktop` must run BEFORE `CoInitialize`**: COM STA pins the
   thread's desktop; switching afterwards fails with ERROR_BUSY(170).
3. **`SendInput` cannot enter the sandbox** (key architectural finding):
   it always injects into the INPUT desktop. From a sandbox-desktop thread
   it fails with ACCESS_DENIED(5); from the default desktop it moves the
   user's real cursor — exactly the red-line behavior. Sandbox input must
   be window messages.
4. **Cross-desktop GDI queries need thread-desktop affinity**:
   `GetWindowRect` against a sandbox window from a default-desktop thread
   returns 0x0.
5. **PAINTSTRUCT is 72 bytes** (x64): handing `BeginPaint` a raw 64-byte
   buffer overwrites 8 heap bytes per WM_PAINT; the symptom is a
   delayed, random access violation (often inside gc or an unrelated
   import). When calling Win32 through ctypes, declare restype/argtypes
   for EVERY function returning a handle or taking a struct pointer —
   otherwise 64-bit handles truncate to 32-bit `c_int` and
   RegisterClass/DefWindowProc crash on garbage.
6. **Edge needs its own profile**: msedge without `--user-data-dir` is
   redirected by the single-instance mechanism to the user's existing
   desktop instance — the launch "succeeds" (a pid comes back) but the
   URL lands in the USER's browser and the sandbox stays empty (45 s of
   confused polling, verified live). The dedicated profile forces a new
   instance that stays in the sandbox.
7. **The `launch()` pid may not be the "real" process**: Win11
   notepad/msedge go through app-execution-alias stubs; the CreateProcess
   pid exits immediately and the real process is a different pid. Manage
   sandbox processes through the Job Object only.
8. **Chromium launchers die on a hidden desktop**: the msedge stub hands
   off to a new process and the handoff chain breaks — the window never
   appears. Wrapping in `cmd.exe /c "..."` keeps a live job member for the
   whole browser lifetime. `sandbox_launch` does this automatically for
   Chromium/Electron commands.
9. **Sandbox Chromium's accessibility tree never activates by itself**:
   nothing ever "reads" it for assistive tech, so UIA scans return an
   empty tree / COMError. `sandbox_launch` appends
   `--force-renderer-accessibility` automatically; even then the first
   scan may return few elements — scan, wait ~3 s, rescan (the wake
   probe's second pass gets the full tree: verified 40 → 166 elements).
10. **SwitchDesktop is FORBIDDEN, not "guarded"**: making the sandbox the
    INPUT desktop would let SendInput work there — and lock the user out
    of their own machine while switched (plus a sandbox-affinity thread
    cannot re-open "Default", err 5). After a live incident the red line
    became absolute: the input desktop is never switched, at any time.
    `SwitchDesktop` now raises `ForegroundViolation` on fetch, and the
    "human-like" layer expresses people-shaped interaction as paced
    messages + a virtual cursor drawn in the projection window.
11. **The IME never interferes** (message mode): WM_CHAR bypasses the IME
    pipeline entirely — no candidate window, no eaten Enter. What used to
    be a hazard of real-keyboard typing in the sandbox is now a structural
    non-issue, because real-keyboard typing in the sandbox does not exist.
12. **UIA rects can exceed the sandbox screen**: element rects come from
    the page layout, not the visible viewport — a link at y=857 on an
    800-high sandbox screen is real but off-screen, and clicking it fails
    with "no sandbox window at (x, y)". Only click rects fully inside the
    sandbox screen; scroll first, or use `sandbox_observe`'s `offscreen`
    state to filter.
13. **Popups are part of the terrain**: a fresh Edge profile triggers the
    crash-restore bubble ("还原页面"), sign-in/sync modals, and the
    "restore pages?" prompt — each eats the first click and shrinks the
    UIA tree to a handful of elements. `sandbox_launch` appends
    `--hide-crash-restore-bubble --no-first-run --no-default-browser-check`
    automatically, and Edge additionally gets `--guest --disable-sync`:
    the SSO sync modal ("同步你的浏览数据") fires even with a dedicated
    profile and non-first-run flags, because Edge auto-signs-in with the
    Windows account — guest mode is the only flag that stops it
    (verified live; the dialog even outranked the main window in
    enumeration order once). Whatever still pops up: observe → dismiss
    via `sandbox_human` → observe again (never trust the first scan).
14. **Plugin install does not register the MCP server** (pit: "installed
    but unknown"): installing the plugin brings the agent + skill, but the
    MCP server needs wiring — this repo now ships `.mcp.json` so Claude
    Code picks up `tandem-agent-mouse serve` at plugin load. If a session
    still reports the server as "Unknown", MCP registration does not hot
    reload: restart the session, or drive the server directly with a
    stdio JSON-RPC probe script. Verify the TOOL LIST, not the docs.
15. **Window pick order is enumeration order, not importance**: a modal
    dialog (Edge's sync dialog, 524×364) can be enumerated BEFORE the
    main window (1280×900) and win a naive `vis[0]` pick — screenshots
    and UIA scans then capture the wrong object. All pickers now rank by
    visible area descending (class preference as tiebreak) through one
    shared `pick_main_window`; keep it that way when adding pickers.
16. **Word's start screen never renders on a hidden desktop**: `WINWORD.EXE`
    with no document shows the Start screen, which stays blank there
    (disabling hardware acceleration does not help — verified live).
    Launch Word with a document instead: create a blank `.docx` (an empty
    file with the right ZIP shape, or `copy NUL` + let Word repair) and
    pass it on the command line; the document window renders fine and
    UIA sees the "Page 1" canvas.
17. **STA-pinned threads cannot switch desktops** (the screenshot bug):
    `run_on_sandbox_thread` initializes COM STA *after* the switch, pinning
    the thread — any further `SetThreadDesktop` inside that worker fails
    with ERROR_BUSY/170. `enumerate_sandbox_windows` therefore skips the
    switch when the calling thread is already on the sandbox desktop
    (GetThreadDesktop == sandbox handle); call it from anywhere.

### B mode v1 (windows-mcp bridge, deprecated but kept)

1. **`selectors` cannot watch Windows pipes**: `DefaultSelector.select()`
   raises WinError 10038 on a child's stdout pipe (pipes are not sockets).
   Use a persistent pump thread + Queue.
2. **`threading.Lock` is not reentrant**: calling `_rpc()` while `start()`
   holds the lock deadlocks; the bridge lock must be an `RLock`.
3. **One reader thread per response steals lines**: concurrent
   `readline()` calls grab each other's responses; reads must converge to
   a single pump thread.

---

## Repository layout

```text
tandem-agent-mouse/
├── src/tandem_agent_mouse/
│   ├── win32/      ctypes native layer + red-line guard (guard.py: raise on fetch)
│   ├── uia/        com_worker (persistent STA thread) + scanner (CacheRequest walk)
│   │               + actions (pattern invocation + message-click fallback)
│   ├── capture/    PrintWindow(PW_RENDERFULLCONTENT) → WGC chain
│   ├── sandbox/    hidden virtual desktop (desktop.py) + message injector
│   │               (input_thread.py) + corner projection (projection.py)
│   │               + manager (manager.py)
│   ├── bridge/     windows-mcp subprocess bridge (legacy B mode, deprecated)
│   ├── core/       Element model, RuntimeId registry, diff engine, batch executor
│   ├── vision/     PNG encoding, grid metadata, visual element wrapping
│   ├── server.py   hand-written stdio JSON-RPC MCP (zero SDK dependencies)
│   └── cli.py      command-line entry
├── tests/          guard / diff / registry / executor / bridge policy / protocol (32)
├── pyproject.toml  hatchling build; dependencies are comtypes + pillow only
├── CHANGELOG.md
└── LICENSE         MIT
```

---

## Tests

```bash
python -m pytest tests/ -q
```

32 tests, all passing. Coverage: the red-line guard (forbidden APIs raise
on fetch), the diff engine, the RuntimeId registry, the batch executor,
the bridge exclusion policy, and the stdio protocol. The sandbox lifecycle
(`stop()` / projection close / hard server kill → zero surviving
processes) is verified separately on a live machine.

---

## Known limitations

Honest edge cases — read before relying on any of this:

- **Minimized windows cannot be captured** (neither WGC nor PrintWindow
  can get content); reported honestly as `WindowMinimized`.
- **Elevated windows need an equally elevated process** for UIA.
- **Chromium web content rejects synthetic message clicks** (pitfall 7);
  inside the sandbox, message clicks do land once the target rect comes
  from UIA (the bilibili flow was verified end-to-end).
- **WGC reports `E_INVALIDARG` on some windows** (cause not identified;
  the fallback chain then fails and reports honestly).
- **Projection frame rate is ~2 fps** (0.5 s interval) — enough to watch
  the agent work, not video-grade smoothness.
- **Windows 10/11 only**; UIA behavior varies with each app's
  accessibility implementation quality.

---

## Reporting issues

Regular bugs / feature requests: file them on the repository's Issues page
with reproduction steps, expected behavior, actual behavior, and the
environment (Windows version, Python version).

## License

MIT — see [LICENSE](LICENSE). Dependencies: comtypes (MIT), Pillow
(MIT-CMU), optional winsdk (MIT). windows-mcp (Apache-2.0) is invoked as a
separate subprocess; its code is not linked.