Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@WinUse Open Notepad and type 'Hello from WinUse'"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
中文说明 · English
A unified desktop-capability facade for AI agents. cua-driver is the engine; WinUse is the steering wheel.
WinUse drives nothing itself — it injects no input, captures no pixels and owns no window handles. It does one thing: translate cua-driver's driver-level primitives into coarse-grained tools that an agent can call safely, cheaply and precisely.
Handing the raw driver to an agent costs you three ways:
Expensive | The default route is "screenshot + vision model" — roughly 2,100–2,700 tokens per look |
Imprecise | On mixed-DPI multi-monitor setups, screenshot pixels and native coordinates must be converted; get the conversion wrong and the click lands somewhere else |
Risky | Any agent can call the driver directly, bypassing approval gates |
WinUse's answer:
Text first — read the Windows UI Automation (UIA) tree by default. A screenshot is an explicit escalation, never the default path. Interactive elements are a minority of the tree (measured ≈38%), so read-only text is trimmed away.
Elements before coordinates — semantic actions (Invoke / ValuePattern) are independent of coordinates, DPI and screen. Pixel clicks are the fallback for self-drawn surfaces only (canvas, video, WebGL, custom-rendered UI).
Act → guard → verify — dangerous actions pass an internal confirmation gate and are written to an audit log. After acting you get a deterministic assertion, not "it seems to have moved".
Related MCP server: screen-use
How the state was fetched | Tokens |
4,804-element window: full snapshot, then trimmed | ≈ 14,788 |
Same window: projected on demand with | ≈ 2,285 |
The saving is a projection, not a lossy truncation: you ask only for the rows you need, and element indexes stay valid inside the same snapshot.
Tool | What it does |
| Text-first state of one window: interactive elements (with |
| Fuzzy element lookup by text — one wrong character still hits |
| Read several elements' values/states in a single round trip |
| Per-monitor physical rectangles, work areas, per-monitor scale and DPI, window ownership |
| Capture to a file (explicit escalation; the only pixel path) |
| Windows built-in OCR — no network, no third-party dependency |
| Whole-disk file search through Everything ( |
Tool | What it does |
| Element-level action: |
| Drag (hold → move → release), element-to-element or by coordinates |
| Batch several interface actions in one call — batch input counts as dangerous |
| Invoke menu items by path, resolved layer by layer |
| Window lifecycle: minimise / maximise / restore / always-on-top / move / resize / close |
| Application lifecycle: launch (without stealing focus) / list / kill |
Tool | What it does |
| Conditional wait: window appears/disappears, title matches, element appears, text appears |
| Deterministic assertions against one window (predicate array, three-state result — |
| OS / CPU / memory / disks / power, plus top processes by memory |
| Session and desktop state: locked or not, secure desktop, which capabilities are usable |
| Which virtual desktop a window belongs to, and whether it is the current one |
| Read the registry (read-only by default; sensitive hives refused) |
| Read/write clipboard — |
| Windows toast notification — dry-run by default |
| Read WinUse's own audit log (read-only) |
Requirement | Necessity | Notes |
Windows 10 / 11 | required | WinUse is Windows-only by design |
Python ≥ 3.10 | required | Zero third-party runtime dependencies — standard library only (Windows API via |
| required | The only driver. Auto-detected from |
| required for OCR / toast | PowerShell 7 ( |
| optional | Fallback channel for notifications only |
"Zero dependencies" is a deliberate design constraint, not a gap: the core and every facade use the standard library only, so any Python runs it and nothing leaks into the environment outside the project.
git clone https://github.com/freehul/winuse.git
cd winuse
python -m winuse doctor # self-check: config -> driver -> one real read-only probe
python -m winuse doctor --no-probe # driver version only, no real callNo installation step is needed to use the CLI. To install it as a package (and to run the tests):
python -m pip install -e ".[test]" # test deps are declared here, never at runtime
python -m pytest -qscripts/verify.sh runs everything in one go (compile -> tests -> warnings-as-errors):
sh scripts/verify.sh # PYTHON='py -3.11' to pick an interpreter
sh scripts/verify.sh --doctor # also run one real read-only self-checkOptional: copy .env.example to .env to override defaults. .env is never committed.
python -m winuse mcp serve # MCP over stdio
python -m winuse mcp tools # list the tool tablepython adapters/hermes/install.py --home <HERMES_HOME>
hermes plugins enable winuse
hermes plugins list # confirm it is enabledSee adapters/hermes/README.md for the deployment model and its two safety gates.
python -m winuse state # text-first state of the foreground window
python -m winuse display # multi-monitor facts
python -m winuse act --help # every action is a dry run until --executeDry-run by default. winuse act prints the command it would send; only --execute (CLI) or execute: true (MCP) really touches the desktop.
Confirmation gate inside the core. Destructive and bulk operations require an explicit confirm, and the gate lives in the core — not in the facades — so a new client cannot forget it.
Audit log. Every executed action is appended to ~/.winuse/audit.jsonl (path and rotation are configurable).
Snapshot binding. An element_index is only valid inside the snapshot_id it came from; actions without the matching snapshot are refused (SNAPSHOT_REQUIRED), which prevents a stale index from clicking the wrong element.
One driver, no second set. A second driver would collide on focus and coordinate conventions and bypass approval gates.
No resident HTTP/SSE server. In-process use goes through a subprocess; external use goes through MCP over stdio — nothing is exposed on a port.
winuse/
config.py configuration (reads the project's own .env, deliberately not process env)
cli.py facade 3: CLI (fallback + troubleshooting)
driver/
cua.py cua-driver wrapper — the only IO boundary
errors.py stable error codes with human-readable messages
adapters/hermes/ Hermes plugin adapter — protocol translation only, no business logic
scripts/ verify.sh, end-to-end smoke, git-hook installer, Windows wrapper
tests/ unit tests; always against a fake driver, never the real desktopDesign contracts, backlog and research notes are maintained privately and are not part of this repository — the code and this README are the published surface.
Unit tests never touch the real driver or the desktop: tests/fake_driver.py and per-case stub drivers stand in for it. Real-machine checks go through python -m winuse doctor and controlled probes.
python -m pytest -q # the only acceptance command
python -m pytest -q -W error # warnings as errors: protocol/resource problems show up here firstIssues and pull requests are welcome. Project rules (zero third-party dependencies, one driver, TDD, data hygiene) live in AGENTS.md; the acceptance entry point is sh scripts/verify.sh — a change is not done until it is green.
Data hygiene is enforced by a commit hook: sh scripts/install_git_hooks.sh enables it once per machine.
MIT — see LICENSE.
This server cannot be deployed
Eyes and hands on real Windows PCs — observe, click, type via Glasswarp API.
Operate Linux, macOS and Windows from your LLM. Every action runs through an auditable allowlist.
Control real Android and iOS devices with LLM agents — tap, swipe, type, automate flows.
Control a real browser from AI agents: navigate, click, type, screenshot, read the page.
