AgentMouse

WeChat workflow failed (2026-11-04): An explicitly authorized live pilot after restart returned real images and kept the physical cursor still, but the click raised WeChat and did not open the QQ Mail conversation. No email was read. Normal WeChat quarantine remains; the previous persistent hang's cause is still unconfirmed. Keep the old installed plugin disabled.

Unreleased source fixes capture lifetime/timeouts, unchanged-image guards, enable/disconnect races, partial accessibility labels, broken worker cleanup and extension release reporting. Audit and scoped test evidence. These fixes are not a WeChat business-compatibility pass or a client upgrade.

Background computer use for AI agents on Windows, with a pointer independent of your real mouse.

中文 · Compatibility and evidence · Releases

Keep working while your agent observes and operates another app. AgentMouse combines a Windows controller, MCP server, reusable Agent Skills and installers for Claude Code, Codex, ZCode, WorkBuddy and DSH. No virtual machine or split screen.

For web tasks, Browser Bridge reads real tab images and sends coordinate-based mouse input through Chrome's debugger API. It uses your existing browser/profile and does not rely on DOM selectors, injected clicks or per-site scripts. Native Windows capture/messages remain available for desktop apps.

Beta: the fusion server passed native Windows control-label-to-background-input integration and ten covered-Chrome desktop checks. Fusion desktop evidence. This does not guarantee every app or real CAPTCHA accepts automation.

Two independent editions

Older downloads are not recommended during the WeChat incident; see the notice above.

Core builds and runs independently. Full calls Core through an adapter. Module boundaries and contributions. Independent keyboard/batch-input compatibility interfaces still return explicit unsupported errors.

Related MCP server: Desktop Controller

Windows-MCP fusion

Fusion combines Windows-MCP system tools/target-scoped controls with AgentMouse background capture/input in one 26-tool MCP server. Native Windows label-to-input integration and ten covered-Chrome desktop cases passed. Fusion setup, capabilities and limits.

Install

Requires Windows x64, Node.js 18+ x64 with npm and .NET Framework. Neither release requires a separate Python/uv installation: Core uses no Python, and Full bundles an isolated runtime. Developers building Full from source need uv. Native covered-window capture requires Windows 10 1903+.

Extract the Windows release ZIP, double-click install.cmd, and select your agents; each package defaults to its own Core/Full engine. From source:

git clone https://github.com/Mint-hfut/AgentMouse.git
cd AgentMouse
.\install.cmd

The source installer installs locked dependencies without lifecycle scripts and compiles native files with Windows' .NET Framework compiler. No Visual Studio/.NET SDK is required. Codex native installation needs its CLI; ZCode uses its bundled plugin CLI in standard install locations or PATH.

.\install.cmd --engine fusion --agent codex,zcode
.\install.cmd --agent all
.\install.cmd --agent dsh --profile web
.\install.cmd --agent codex --mode mcp
.\install.cmd --agent codex --browser none

Codex/ZCode default to native plugin installation, verified through their installed-plugin lists. Claude Code, WorkBuddy and DSH currently receive MCP + skills, which does not create a plugin card. --mode mcp selects that mode explicitly. --browser none skips browser-file preparation.

Refresh/restart the selected agents after installation. The native plugin is agentmouse@agentmouse.

Connect your existing Chrome once

The installer prepares a paired extension directory and prints its path, normally ~/.agentmouse/browser/extension. It does not change Chrome settings, profiles or enterprise policies.

1. In Chrome, open chrome://extensions, enable Developer mode and choose Load unpacked yourself.

2. Select the generated directory printed by the installer, not the unpaired source folder.

3. Review the debugger permission. It provides page images and mouse input. No remote debugging port, new browser profile or disabled GPU is needed for normal use.

4. Authorize the current input task; the agent can call Full pointer_open(allow_input=true) or Core pointer_open(allowInput=true) without repeating that confirmation. Select a browser: entry from the window list for web work.

If enterprise policy blocks this, stop and consult your administrator. The installer does not bypass it. Browser preparation can also be run separately:

node agentmouse.mjs browser-setup
node agentmouse.mjs browser-status
node agentmouse.mjs browser-stop

The controller UI is currently Chinese. Chrome may display its standard debugging indicator. Use one paired browser profile and one controlling agent at a time.

Native-only mouse and capture capabilities

The following 11-tool reference is for native-only mode. Fusion tool names and target/token arguments are documented in fusion.md.

There is no permanent target binding. Every click/drag/scroll explicitly supplies the intended windowId and its fresh observation token; mismatches are rejected. Supplying a window ID defaults to real window capture, never a crop of the covering foreground app. A desktop overview has no fixed target; any input still needs an explicit intended window ID. Reobserve another target to switch apps/tabs.

Query pointer_list_apps before launching an app: missing visible windows can mean a running tray/minimized app. Hidden/minimized windows are discoverable but cannot be captured or clicked; AgentMouse does not automatically restore them. pointer_inspect_window adds Windows-MCP-style semantic context using built-in .NET UIAutomation, without a Python runtime or foreground input tools. Some custom apps expose only window chrome.

Native capture uses one isolated helper per screenshot and closes capture resources before returning pixels. Timeout or pause can stop that helper directly. After native input, unchanged pixels or an unchanged/older timestamp returns inputAllowed=false; stop rather than repeating clicks. A changed image still does not prove a business outcome.

Follow the returned coordinate space: native captures use physical desktop pixels; browser images use viewport CSS pixels. Both return image origin and scaling metadata:

x = screenshot.x + imageX * screenshot.scaleX
y = screenshot.y + imageY * screenshot.scaleY

Observations expire after 15 seconds and are consumed by input. Verify the actual image/state afterward; inputDelivered is not business success. Cancellation/failure releases buttons where the channel remains available; an unconfirmed release blocks more input. Forced termination cannot guarantee all cleanup.

Native messages cannot guarantee inactive Chromium windows remain behind: Chrome can activate itself during mouse presses. Those inputs are blocked and require Browser Bridge. Native app compatibility still varies; minimized/cloaked windows, independent keyboard input and cross-app system drag-and-drop are not promised. Touch is experimental and never a background fallback.

Tested scope

The real extension path has passed local HTML checks for left/right clicks, wheel, native range slider, curved puzzle-style drag, long press, double click, ordered point selection, HTML5 drag/drop, and release after mid-drag pause. Assertions check page state and captured pixels. These are local interaction checks, not third-party CAPTCHA acceptance; server risk checks can still require a person.

Desktop non-interference, actual-client native loading and component tests are separate evidence categories. The validation record preserves failures and pending checks.

Configuration and removal

Runtime: %LOCALAPPDATA%\AgentMouse\0.7.0-beta.3 (fusion adds a -fusion suffix; physical path resolved for packaged Windows hosts). Browser pairing: ~/.agentmouse/browser, with a per-user local token and authenticated loopback-only service. Generated secrets/private profiles stay outside the repo. Existing settings are retained with backups under the runtime's backups/.

MCP-only paths: Claude ~/.claude.json, Codex ~/.codex/config.toml, ZCode ~/.zcode/cli/config.json, WorkBuddy ~/.workbuddy/mcp.json, DSH ~/.dsh/profiles/<profile>/cordis.patch.yml. Codex native upgrades relocate only AgentMouse's own marketplace when the versioned source changes. Native installer failures report possible partial completion; MCP-only settings writes support rollback.

Use --dry-run to preview after npm ci --ignore-scripts. --home isolates a home for testing; --install-dir selects the runtime. Do not retain a standalone MCP registration alongside the native plugin's same server. Remove native plugins in the host plugin manager; stop the browser bridge and remove its extension before deleting runtime/pairing files. Restoring whole backups also reverts later unrelated edits.

Build and validation

npm ci --ignore-scripts
npm run build
npm test
npm run test:state
node scripts/test-clients.mjs

The first four commands do not perform desktop input. test-clients reads installed Codex/ZCode payloads and checks their declared server/SDK path; it is not a GUI plugin-browser test. CI runs Node 18/24, isolated installer checks and a real extension/component test with pinned Chrome for Testing. The latter is headless and does not claim desktop non-interference.

scripts/build-release.ps1 builds a complete ZIP and SHA-256 file; scripts/test-release.ps1 verifies its extracted MCP installation. See adapter references and interactive test instructions.

Apache-2.0. Not affiliated with the supported agent/browser products.