tandem-agent-mouse

Background-first desktop automation for Windows as an MCP server: the agent sees and drives your apps — without ever disturbing you.

Lets Claude (or any MCP client) read any window's UIA accessibility tree, take background screenshots, and perform clicks and typing. A mode works on the user's desktop in the background — it never moves the real cursor or steals focus, and the forbidden APIs raise on fetch. B mode runs the agent's apps on a hidden second desktop that the user watches through a corner projection window; input is injected as window messages and the sandbox's whole process tree is guaranteed orphan-free by a Job Object.

English · 简体中文

Version: 1.0.0b1 (beta) | Platform: Windows 10/11 | Tests: 32

Red lines (enforced in code, not by convention)

Every red line below is structurally enforced by guard.py — it is not a prompt-level promise:

• A mode never touches the user's desktop state. SetForegroundWindow, SetCursorPos, SendInput, keybd_event, mouse_event, AttachThreadInput, SetFocus raise ForegroundViolation on fetch in the parent process — not a runtime warning, a missing function pointer.

• The input desktop is NEVER switched — at any time. SwitchDesktop is blocked by the same guard. Making the sandbox the input desktop would let a "real cursor" work there, but it locks the user out of their own machine while switched. Sandbox "human-like" input is instead paced message injection plus a virtual cursor drawn inside the projection window — pixels in a 480 px corner window, never the user's pointer.

• Total background failure returns needs_foreground. The user decides whether to switch to B mode; the server never switches on its own.

• B mode input is window-message injection only (PostMessage into the sandbox's windows). Measured on Win11 26200: SendInput always injects into the INPUT desktop (Default while the user works) — from a sandbox thread it fails with ACCESS_DENIED, and from the default desktop it would move the user's real cursor. Message injection makes "never touches the user's cursor" structurally true.

• Zero orphaned sandbox processes. Every sandbox process starts CREATE_SUSPENDED, joins a KILL_ON_JOB_CLOSE Job Object, then resumes. Closing the projection window, set_mode A, graceful server exit, a hard server kill — all four paths terminate the whole tree (kernel-enforced).

Related MCP server: Computer Use MCP

Why tandem-agent-mouse?

Existing desktop-automation approaches force a choice: pure vision (screenshot + coordinate clicks — slow and brittle) or foreground takeover (steals the mouse and focus — the user must stop working). This repository provides a third path:

The recognition layer is UIA, not pixels. One unified element format (shared by the UIA and visual paths):

{
  "id": 17, "type": "button", "name": "OK", "value": null,
  "state": ["enabled", "offscreen"],
  "rect": [120, 80, 64, 24],
  "actions": ["invoke", "click", "toggle"],
  "depth": 3, "source": "uia"
}

Priority chain: UIA CacheRequest full-tree scan → (empty tree) wake probe + retry → (still failing) PrintWindow(PW_RENDERFULLCONTENT) → (blank) Windows Graphics Capture → honest errors. Screenshots return MCP ImageContent plus grid metadata; the client reads the image, targets a point, and sends it back through act's click_at.

A/B modes, one tool surface. A mode is the default: background UIA reads, background captures, batch actions that return only the diff. B mode is opt-in: hidden sandbox desktop + corner projection — the user and the agent each operate their own world, neither fighting for the mouse, both able to watch (and nudge) the other.

How it works

flowchart LR
    subgraph CLIENT["MCP client (Claude, ...)"]
        T["list_windows / observe / act /<br/>screenshot / set_mode /<br/>sandbox_*"]
    end
    T -->|stdio JSON-RPC<br/>zero SDK| SRV["server.py"]

    subgraph GUARD["win32/guard.py — red-line guard"]
        G["SetForegroundWindow / SendInput /<br/>SetCursorPos / keybd_event<br/>raise on fetch"]
    end

    SRV -->|"A mode"| UIA["uia/ persistent STA thread<br/>CacheRequest single-pass scan<br/>RuntimeId-stable identity"]
    UIA --> ACT["core/executor batch run<br/>invoke/toggle/set_value<br/>+ diff"]
    ACT -->|honest failure| NF["needs_foreground"]

    SRV -->|"B mode"| SB["sandbox/"]
    SB --> D["desktop.py<br/>CreateDesktopW hidden desktop<br/>+ Job Object membership"]
    SB --> IN["input_thread.py<br/>PostMessage injection<br/>(no cursor touched)"]
    SB --> PJ["projection.py<br/>topmost corner window<br/>frame pump 0.5s/frame"]
    PJ -->|"user clicks projection"| IN2["mapped back to sandbox<br/>coords, forwarded"]
    PJ -->|"user clicks X"| STOP["stop()<br/>TerminateJobObject<br/>whole tree dies"]

Three things the diagram makes explicit:

• The guard is the bottom layer: any code reaching for a forbidden API passes through guard.py first.

• A-mode failures are honest: a UIA pattern that will not resolve, a message click that Chromium refuses — reported as needs_foreground, never silently degraded into foreground theft.

• B-mode shutdown is kernel-guaranteed: the moment the projection window closes, the Job Object closes with it and every sandbox process — including the real process behind an alias stub — dies.

Installation

cd "Tandem Agent Mouse"
uv pip install --system -e .
# Base (PrintWindow captures); add WGC for the Electron blank-frame fallback:
uv pip install --system winsdk
# Tests
python -m pytest tests/ -q
# stdio MCP server
python -m tandem_agent_mouse serve

Registering in Claude Code / Claude Desktop

{
  "mcpServers": {
    "tandem-agent-mouse": {
      "command": "python",
      "args": ["-m", "tandem_agent_mouse", "serve"]
    }
  }
}

Other MCP clients (ZCode and friends) take the same configuration.

Quick start

Talk to the client in natural language. Typical flows:

A mode (background, undisturbed):

"Write something in Notepad"  -> observe finds the edit area -> act set_value
"Show me this window"         -> screenshot returns image + grid + click_at usage

B mode (sandbox + projection, you watch it work):

sandbox_mode "sandbox" -> the "Tandem Agent Mouse Sandbox" corner window appears
sandbox_launch Edge    -> Edge opens on the hidden desktop, visible in the projection
sandbox_click/type     -> message-injected search, navigation, playback
sandbox_human          -> paced, human-like input; virtual cursor drawn in the window
close the projection   -> the sandbox tears down, zero processes left

Live-verified example: Edge on the sandbox -> bilibili -> search "Luo Xiaohei" -> open the Luo Xiaohei Zhanji playback page — the user's cursor never moved and no focus was stolen at any point.

MCP tools

Two parallel tool families — no A/B letters to remember:

• background_* — work on the USER's desktop. Never touches the cursor or focus; forbidden APIs raise on fetch.

• sandbox_* — work in the hidden sandbox desktop. The projection window shows it; all input is message-injected.

Legacy names (list_windows, observe, act, screenshot, set_mode) remain as routing aliases for old clients but are no longer advertised.

act executes N steps per round trip and returns only the added/removed/changed elements; the settle interval between steps is configurable (default 350 ms).

B mode v2: sandbox + corner projection

1. Hidden sandbox desktop: CreateDesktopW creates a second desktop; the agent's programs (e.g. Edge) open there via STARTUPINFO.lpDesktop, invisible to the user's desktop.

2. Corner projection window: a topmost bottom-right window (title "Tandem Agent Mouse Sandbox — 关闭=结束沙箱") fed by a frame pump thread that PrintWindows the sandbox's main window and blits it; user clicks inside the projection map back to real sandbox coordinates and are forwarded. A virtual-cursor overlay draws the agent's input steps (trail, cursor ring, click ripples) on top of the mirror, plus a status bar. Closing the projection is the user's explicit decision to end the sandbox; the pump detects the close and tears everything down.

3. Message-injected input: sandbox_click/type PostMessage WM_MOUSEMOVE/LBUTTONDOWN/CHAR directly to sandbox windows. No cursor moves, no focus changes — the user keeps working while the agent works "next door".

Lifecycle (no orphaned sandbox processes, by construction)

• Job Object membership: sandbox_launch starts the process CREATE_SUSPENDED, assigns it to a KILL_ON_JOB_CLOSE job, then resumes it — the process cannot spawn children before it joins the job, so the whole tree is inside the net.

• Four shutdown paths, all killing every sandbox process:

  1. user closes the projection window → pump detects → stop() → TerminateJobObject;

  2. the agent calls set_mode A → stop();

  3. graceful server exit (stdio EOF) → atexit.register(_sandbox.stop);

  4. hard server kill (TerminateProcess/crash) → atexit never runs, but the dying process closes the job handle and KILL_ON_JOB_CLOSE fires in the kernel.

• Verified on a live machine: after stop(), after projection close, and after taskkill /F on the server — zero sandbox processes remain.

Chelate Shell integration

This repository also ships as the tandem-agent-mouse skill of Chelate Shell (stage=infrastructure): skills/tandem-agent-mouse/run.py is a thin adapter with zero copied logic — everything routes into this package's modules; missing comtypes/pillow degrade to an UNVERIFIED gate instead of crashing the pipeline.

Field-tested pitfalls (all verified on real machines; do not re-learn them)

Recognition layer (Windows 11 + Python 3.12)

1. CacheRequest.TreeScope must include the element itself (7 = element+children+descendants). Passing 4 (subtree only) fails with E_INVALIDARG.

2. Pattern availability flags have no CachedIs*PatternAvailable property; read GetCachedPropertyValue(30031..30044).

3. Do not put Value(30012) in the CacheRequest: Chromium pane elements reject the property, truncating the whole cached walk to one level. Read Value per element with GetCachedPropertyValue.

4. All UIA COM calls run on one persistent STA thread (com_worker). Per-call threads + CoInitialize/CoUninitialize make Chromium providers return NULL pattern pointers on the second call.

5. IsEnabled=30010, BoundingRectangle=30001 — 30006 is AcceleratorKey; a wrong property ID only explodes when that element is read.

6. Chromium's accessibility tree is lazily activated: wake it with ElementFromHandle (sends WM_GETOBJECT), wait ~0.5 s, rescan; a minimized/restored app may sleep again.

7. Chromium background clicks: the browser chrome (title-bar buttons) accepts WM_LBUTTON*; web content does not (the renderer process owns input) — web content goes through UIA Invoke, and failure reports needs_foreground honestly.

8. PrintWindow(PW_RENDERFULLCONTENT) captures most Electron/Chromium windows; the occasional all-blank result falls through to WGC (optional winsdk dependency).

9. Cross-round element identity uses the UIA RuntimeId (stable across scans on Chromium in practice).

Sandbox (Windows 11 26200)

1. CreateDesktopW lives in user32: old documentation says kernel32; on Win11 it is user32.

2. SetThreadDesktop must run BEFORE CoInitialize: COM STA pins the thread's desktop; switching afterwards fails with ERROR_BUSY(170).

3. SendInput cannot enter the sandbox (key architectural finding): it always injects into the INPUT desktop. From a sandbox-desktop thread it fails with ACCESS_DENIED(5); from the default desktop it moves the user's real cursor — exactly the red-line behavior. Sandbox input must be window messages.

4. Cross-desktop GDI queries need thread-desktop affinity: GetWindowRect against a sandbox window from a default-desktop thread returns 0x0.

5. PAINTSTRUCT is 72 bytes (x64): handing BeginPaint a raw 64-byte buffer overwrites 8 heap bytes per WM_PAINT; the symptom is a delayed, random access violation (often inside gc or an unrelated import). When calling Win32 through ctypes, declare restype/argtypes for EVERY function returning a handle or taking a struct pointer — otherwise 64-bit handles truncate to 32-bit c_int and RegisterClass/DefWindowProc crash on garbage.

6. Edge needs its own profile: msedge without --user-data-dir is redirected by the single-instance mechanism to the user's existing desktop instance — the launch "succeeds" (a pid comes back) but the URL lands in the USER's browser and the sandbox stays empty (45 s of confused polling, verified live). The dedicated profile forces a new instance that stays in the sandbox.

7. The launch() pid may not be the "real" process: Win11 notepad/msedge go through app-execution-alias stubs; the CreateProcess pid exits immediately and the real process is a different pid. Manage sandbox processes through the Job Object only.

8. Chromium launchers die on a hidden desktop: the msedge stub hands off to a new process and the handoff chain breaks — the window never appears. Wrapping in cmd.exe /c "..." keeps a live job member for the whole browser lifetime. sandbox_launch does this automatically for Chromium/Electron commands.

9. Sandbox Chromium's accessibility tree never activates by itself: nothing ever "reads" it for assistive tech, so UIA scans return an empty tree / COMError. sandbox_launch appends --force-renderer-accessibility automatically; even then the first scan may return few elements — scan, wait ~3 s, rescan (the wake probe's second pass gets the full tree: verified 40 → 166 elements).

10. SwitchDesktop is FORBIDDEN, not "guarded": making the sandbox the INPUT desktop would let SendInput work there — and lock the user out of their own machine while switched (plus a sandbox-affinity thread cannot re-open "Default", err 5). After a live incident the red line became absolute: the input desktop is never switched, at any time. SwitchDesktop now raises ForegroundViolation on fetch, and the "human-like" layer expresses people-shaped interaction as paced messages + a virtual cursor drawn in the projection window.

11. The IME never interferes (message mode): WM_CHAR bypasses the IME pipeline entirely — no candidate window, no eaten Enter. What used to be a hazard of real-keyboard typing in the sandbox is now a structural non-issue, because real-keyboard typing in the sandbox does not exist.

12. UIA rects can exceed the sandbox screen: element rects come from the page layout, not the visible viewport — a link at y=857 on an 800-high sandbox screen is real but off-screen, and clicking it fails with "no sandbox window at (x, y)". Only click rects fully inside the sandbox screen; scroll first, or use sandbox_observe's offscreen state to filter.

13. Popups are part of the terrain: a fresh Edge profile triggers the crash-restore bubble ("还原页面"), sign-in/sync modals, and the "restore pages?" prompt — each eats the first click and shrinks the UIA tree to a handful of elements. sandbox_launch appends --hide-crash-restore-bubble --no-first-run --no-default-browser-check automatically, and Edge additionally gets --guest --disable-sync: the SSO sync modal ("同步你的浏览数据") fires even with a dedicated profile and non-first-run flags, because Edge auto-signs-in with the Windows account — guest mode is the only flag that stops it (verified live; the dialog even outranked the main window in enumeration order once). Whatever still pops up: observe → dismiss via sandbox_human → observe again (never trust the first scan).

14. Plugin install does not register the MCP server (pit: "installed but unknown"): installing the plugin brings the agent + skill, but the MCP server needs wiring — this repo now ships .mcp.json so Claude Code picks up tandem-agent-mouse serve at plugin load. If a session still reports the server as "Unknown", MCP registration does not hot reload: restart the session, or drive the server directly with a stdio JSON-RPC probe script. Verify the TOOL LIST, not the docs.

15. Window pick order is enumeration order, not importance: a modal dialog (Edge's sync dialog, 524×364) can be enumerated BEFORE the main window (1280×900) and win a naive vis[0] pick — screenshots and UIA scans then capture the wrong object. All pickers now rank by visible area descending (class preference as tiebreak) through one shared pick_main_window; keep it that way when adding pickers.

16. Word's start screen never renders on a hidden desktop: WINWORD.EXE with no document shows the Start screen, which stays blank there (disabling hardware acceleration does not help — verified live). Launch Word with a document instead: create a blank .docx (an empty file with the right ZIP shape, or copy NUL + let Word repair) and pass it on the command line; the document window renders fine and UIA sees the "Page 1" canvas.

17. STA-pinned threads cannot switch desktops (the screenshot bug): run_on_sandbox_thread initializes COM STA after the switch, pinning the thread — any further SetThreadDesktop inside that worker fails with ERROR_BUSY/170. enumerate_sandbox_windows therefore skips the switch when the calling thread is already on the sandbox desktop (GetThreadDesktop == sandbox handle); call it from anywhere.

B mode v1 (windows-mcp bridge, deprecated but kept)

1. selectors cannot watch Windows pipes: DefaultSelector.select() raises WinError 10038 on a child's stdout pipe (pipes are not sockets). Use a persistent pump thread + Queue.

2. threading.Lock is not reentrant: calling _rpc() while start() holds the lock deadlocks; the bridge lock must be an RLock.

3. One reader thread per response steals lines: concurrent readline() calls grab each other's responses; reads must converge to a single pump thread.

Repository layout

tandem-agent-mouse/
├── src/tandem_agent_mouse/
│   ├── win32/      ctypes native layer + red-line guard (guard.py: raise on fetch)
│   ├── uia/        com_worker (persistent STA thread) + scanner (CacheRequest walk)
│   │               + actions (pattern invocation + message-click fallback)
│   ├── capture/    PrintWindow(PW_RENDERFULLCONTENT) → WGC chain
│   ├── sandbox/    hidden virtual desktop (desktop.py) + message injector
│   │               (input_thread.py) + corner projection (projection.py)
│   │               + manager (manager.py)
│   ├── bridge/     windows-mcp subprocess bridge (legacy B mode, deprecated)
│   ├── core/       Element model, RuntimeId registry, diff engine, batch executor
│   ├── vision/     PNG encoding, grid metadata, visual element wrapping
│   ├── server.py   hand-written stdio JSON-RPC MCP (zero SDK dependencies)
│   └── cli.py      command-line entry
├── tests/          guard / diff / registry / executor / bridge policy / protocol (32)
├── pyproject.toml  hatchling build; dependencies are comtypes + pillow only
├── CHANGELOG.md
└── LICENSE         MIT

Tests

python -m pytest tests/ -q

32 tests, all passing. Coverage: the red-line guard (forbidden APIs raise on fetch), the diff engine, the RuntimeId registry, the batch executor, the bridge exclusion policy, and the stdio protocol. The sandbox lifecycle (stop() / projection close / hard server kill → zero surviving processes) is verified separately on a live machine.

Known limitations

Honest edge cases — read before relying on any of this:

• Minimized windows cannot be captured (neither WGC nor PrintWindow can get content); reported honestly as WindowMinimized.

• Elevated windows need an equally elevated process for UIA.

• Chromium web content rejects synthetic message clicks (pitfall 7); inside the sandbox, message clicks do land once the target rect comes from UIA (the bilibili flow was verified end-to-end).

• WGC reports E_INVALIDARG on some windows (cause not identified; the fallback chain then fails and reports honestly).

• Projection frame rate is ~2 fps (0.5 s interval) — enough to watch the agent work, not video-grade smoothness.

• Windows 10/11 only; UIA behavior varies with each app's accessibility implementation quality.

Reporting issues

Regular bugs / feature requests: file them on the repository's Issues page with reproduction steps, expected behavior, actual behavior, and the environment (Windows version, Python version).

License

MIT — see LICENSE. Dependencies: comtypes (MIT), Pillow (MIT-CMU), optional winsdk (MIT). windows-mcp (Apache-2.0) is invoked as a separate subprocess; its code is not linked.