computer-use-mcp
README.md
# computer-use-mcp
**Give your coding agent hands.** A free, open-source MCP server that lets a chat agent drive **any
Windows desktop app** through window-relative, token-cheap tools - built for VS Code Chat agents, but
it speaks plain MCP stdio and works with any host.
**Vision is powered by the DeepSeek vision model.** Screenshots never enter the model context by
default; when the agent must actually *see* something, the `look` tool asks DeepSeek one narrow
question (cached and opt-in - a key enables it, every other tool works without one).
[](https://github.com/belaszalontai/computer-use-mcp/actions/workflows/ci.yml)
[](https://github.com/belaszalontai/computer-use-mcp/actions/workflows/windows-zip-release.yml)
[](LICENSE)
[](#requirements)
[](#option-b---from-source)
[](https://api-docs.deepseek.com/)
> [!WARNING]
> **Use at your own risk.** This server hands a chat agent real control over your desktop: it can
> click, type, paste, drag and scroll **inside the target program**, open dialogs, and change or
> delete whatever that program lets it change - at machine speed, whether or not you are watching.
> A confused, manipulated or simply over-eager agent can therefore cause real damage (unsaved work,
> wrong data, messages sent, purchases made, and so on).
>
> - Run it in a session or VM you can afford to experiment on; keep backups of anything valuable and
> keep the first runs supervised.
> - Read and approve the tool calls; avoid autopilot / allow-all mode on a daily-driver desktop.
> - Keep the kill switch at hand: hold **Ctrl+Alt+Q** or drop a `STOP` file (see [Safety](#safety)).
> - This software is provided **"as is", without warranty of any kind** - see [LICENSE](LICENSE).
## Why it is different
Most "computer use" setups burn tokens on screenshots and guess absolute screen coordinates. This
server inverts that:
- **Window-relative coordinates, always.** Every point is `(x, y)` inside the *target window*
(0,0 = its top-left), in the same pixel space as the screenshots. No monitor arithmetic, no DPI
surprises, no guessing which display is which.
- **Screenshots are free.** `screenshot` saves a PNG and returns a `shot_id`, the view size and a
`changed` percentage. No image ever enters the model context unless `look` is called.
- **Vision is opt-in, cached and DeepSeek-powered.** `look` is the only costly tool: one narrow
question to the DeepSeek vision model, optional region crop, results cached by image+question.
Deterministic checks (`read_pixel`, `read_text`, `ui_map`) answer most questions for free.
- **The server waits, not the model.** `wait_until` blocks server-side on elements, window titles,
pixels or a stable region - one tool call instead of a poll loop.
- **Structured UI first, pixels last.** `ui_map` / `find_control` / `click_control` / `set_value`
drive named UI Automation controls; freehand strokes are the documented last resort.
- **Safety rails on by default.** Input is refused unless the target window is in front, every
failure carries a stable error code and a `suggested_recovery` line, hold **Ctrl+Alt+Q** (or drop a
`STOP` file) to abort, and every action is logged to `logs/actions.jsonl`.
Measured on DeepSeek's API, 60 vision requests cost on the order of **$0.01** (≈44k tokens) - and a
typical drawing session needs only ~5% of its calls to be vision; everything else is deterministic.
## What is in the box
| Piece | Path | Purpose |
| --- | --- | --- |
| MCP server | `server.py`, `actions.py`, `vision.py` | The 23 tools listed below. |
| Custom agent | `.github/agents/computer-use.agent.md` | A VS Code agent tuned for GUI work: recon-first, cheap checks before vision, commit/deselect rituals, failure playbook. |
| Skills | `.github/skills/paint-drawing`, `.github/skills/blender` | Measured field handbooks for specific apps. The agent loads them on demand - and anyone can contribute more. |
The agent and the skills are **part of the product**: drop them into your project's `.github/` folder
and your own agent inherits the hard-won rituals (selection/floating-state handling, dialog flows,
UIA-first routing, batching for speed).
See it in action: three Paint scenes drawn end-to-end by an agent with the bundled skill -
[demo 1](docs/images/paint-demo-1.png) · [demo 2](docs/images/paint-demo-2.png) ·
[demo 3](docs/images/paint-demo-3.png).
## Requirements
- Windows 10/11 x64 (the server uses Win32, GDI and UI Automation).
- **Release ZIP:** nothing else - `computer-use-mcp.exe` is self-contained.
- **From source:** Python 3.11+ (3.13 recommended).
- **Optional:** a DeepSeek API key for the `look` vision tool. Without it, `look` reports
`VISION_DISABLED` and every other tool works unchanged.
## Quick start
### Option A - Windows release ZIP (recommended)
1. Download `computer-use-mcp-<version>-windows-x64.zip` from
[Releases](../../releases) and extract it anywhere.
2. Add the server to VS Code. Copy `mcp.example.json` into your project's `.vscode/mcp.json` (or
merge the `computer` entry) and fix the two paths:
```json
{
"servers": {
"computer": {
"type": "stdio",
"command": "C:\\path\\to\\computer-use-mcp.exe",
"envFile": "C:\\path\\to\\.env"
}
}
}
```
3. Copy the agent and skills into your workspace so the chat agent can use them:
```powershell
xcopy /E /I <extracted>\.github\agents .github\agents
xcopy /E /I <extracted>\.github\skills .github\skills
```
4. Restart VS Code, pick the **computer-use** agent in the chat picker, approve the `computer/*`
tools (or set pre-approval via **Chat: Manage Tool Approval**), and ask:
> Target: Notepad. Type "Hello from the agent" into the document, then read it back and confirm.
### Option B - from source
```powershell
py -3.13 -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements.txt
```
`.vscode/mcp.json` in this repository already registers the server with the venv interpreter.
In VS Code: **MCP: List Servers → computer → Start**, then use the **computer-use** agent.
Optional vision setup: copy `.env.example` to `.env` and fill in your key.
```ini
DEEPSEEK_API_KEY=sk-...
```
Verify the install without moving the mouse:
```powershell
.\.venv\Scripts\python.exe scripts\smoke_test.py
```
## Tools
| Group | Tools |
| --- | --- |
| Window | `list_windows`, `set_target_window`, `focus_target` |
| Image | `screenshot` (cursor crosshair, `region=`, `title=`), `read_pixel`, `look` |
| Mouse | `click`, `move`, `drag`, `draw_path`, `scroll` |
| Keyboard | `type_text`, `press` |
| UI Automation | `ui_map`, `find_control`, `click_control` (semantic invoke/select/toggle, click fallback), `set_value`, `fill_form`, `read_text` |
| Waiting | `wait`, `wait_until` (element/window/pixel/pixel_changed/stable) |
| Batch | `run` (many steps in one call, `auto_focus`, end capture) |
| Misc | `cursor_position` |
Mutating tools answer with `[outcome: executed|verified|partial|...]` and, on failure, a
`suggested_recovery` line (for example `focus_target` after a focus steal).
### Cost ladder (cheapest question first)
1. `read_pixel(x, y)` - exact colour at a window-relative point (free).
2. `read_text` / `ui_map` - text and named controls without vision (free).
3. `screenshot` - `changed:` % versus the previous frame (free).
4. `look` - DeepSeek vision, only for "what is actually drawn" questions; narrow question, small region.
## Agent and skills
The `.github/` folder is the human side of the product:
- **Agent** (`.github/agents/computer-use.agent.md`): the operating manual. Least-fragile-route
ladder, recon pass for unknown apps, dialog discipline, live/floating object commit rituals,
batching for speed, and a symptom → action failure table. It pins the DeepSeek V4.1 Flash model for
BYOK setups; edit the frontmatter to use a different model.
- **Skills** (`.github/skills/<name>/SKILL.md`): app-specific measured handbooks. The agent reads the
matching skill *before* touching the GUI. Contributing one is only Markdown - see
[CONTRIBUTING.md](CONTRIBUTING.md).
## How it works
```mermaid
flowchart LR
A[Chat agent] -- "MCP stdio (JSON-RPC)" --> B[computer-use-mcp]
B --> C[Win32: windows, GDI capture, input]
B --> D[UI Automation: named controls]
B -- "only look" --> E[DeepSeek vision - optional, cached]
B --> F[(logs/actions.jsonl)]
```
- One synchronous action at a time: a middleware lock serializes tool calls so mouse/keyboard input
never interleaves.
- The target window is re-resolved on every call (bounds, DPI, foreground state), so
`expected_shot_id` can refuse an action when the screen changed under you.
- UI Automation runs on one long-lived COM thread (COM wrappers are finalized there - a Windows
quirk that otherwise crashes long sessions).
## Safety
The warning above is not boilerplate: this tool drives a real program with real data. Before the
first longer run, make sure the target can tolerate mistakes (a scratch document, versioned files or
backups), and stay present for the first sessions.
- **Abort:** hold **Ctrl+Alt+Q** during any action, or create a `STOP` file in the data directory
(`%LOCALAPPDATA%\computer-use-mcp` for the packaged build, the repository root for source runs).
Delete it to resume.
- **Preflight:** input is refused with `ERROR[not_foreground]` when the target window is not in
front; app-owned dialogs and menus count as "in front".
- **Bounds:** points outside the target window are rejected; drags always release the mouse button,
even mid-abort.
- **Audit:** every action is appended to `logs/actions.jsonl`.
- **API key:** lives only in your local `.env` (git-ignored, scanned in CI). Never commit it.
- Do not run the agent in autopilot/allow-all mode on a workstation with valuable data - use a VM or
a dedicated session for untrusted content. See [SECURITY.md](SECURITY.md).
## Limits (v1)
- The capture copies a screen region: keep the target window visible (don't cover it).
- Elevated (UAC) windows cannot receive synthetic input (UIPI).
- One tool call at a time (synchronous); use `run(steps)` to batch known sequences.
- Single target window at a time; switching targets re-anchors all coordinates.
## Contributing
This project is free and grows through contributions of two kinds - both are equally welcome:
- **Build a skill** for an app you know well. No server code required, just a `SKILL.md` with
measured facts and a failure playbook. This is the highest-leverage contribution.
- **Improve the server** - new tools, better UI Automation coverage, reliability fixes. Plain Python,
small surface, documented conventions.
See [CONTRIBUTING.md](CONTRIBUTING.md) for the development setup, the skill quality bar and the PR
flow. Bug reports with a tool-call log excerpt from `logs/actions.jsonl` are gold.
## Development and releases
```powershell
# checks (also run in CI)
.\.venv\Scripts\python.exe -m ruff check .
.\.venv\Scripts\python.exe scripts\check_no_secrets.py
.\.venv\Scripts\python.exe scripts\mcp_handshake.py -- .\.venv\Scripts\python.exe server.py
# Windows release build (creates AND handshake-tests the executable)
.\build-windows-zip.ps1 -Clean
```
The version lives in `version.py` (single source of truth); releases are driven by a GitHub release:
publishing `v<version>` runs the `windows-zip-release` workflow, which verifies the tag against
`version.py`, builds the ZIP, attaches it to the release and confirms the asset. Maintainers: follow
[RELEASE-CHECKLIST.md](RELEASE-CHECKLIST.md).
```
server.py MCP tools (mcp SDK: MCPServer)
actions.py Windows primitives: windows, capture, input, UI Automation
vision.py optional DeepSeek vision call with caching
version.py package version (single source of truth)
scripts/ dev + release tooling (smoke tests, ZIP writer, secret scan)
.github/ agent, skills, workflows
```
## Roadmap
- `click_element(id)` grounded on the latest vision analysis.
- PrintWindow-based capture for occluded windows.
- Macro recording: replayable, parameterized step sequences on top of `run`.
## License
[Apache-2.0](LICENSE). Free for personal and commercial use; contributions are accepted under the
same terms.
This server cannot be deployed
Maintenance
ActivityMaintained
ResponsivenessNo issues