Skip to main content
Glama
README.md
# computeruse

computeruse is a computer-use harness for Windows. It exposes desktop apps and browsers to AI
agents (Claude, Codex, Kimi) as MCP tools, and confines the agent to its own virtual desktop or
monitor so that a person can keep using the same PC.

## Features

- Reads windows through UI Automation and acts with messages posted to the window, without
  moving the person's cursor or using their keyboard or clipboard.
- A territory for the agent: a virtual desktop, a monitor, or half of a single monitor. The
  agent can observe any window and can act only inside its territory or on windows that the
  person hands over.
- Browser control for Chrome and Edge: tabs, forms and page reads.
- Drivers for Office documents and for Java (Swing/AWT) apps through the Java Access Bridge.
- A sign-off gate for actions that send, delete, pay or grant permissions.
- A password store in Windows Credential Manager. Forms take a `{secret:name}` placeholder, and
  the model does not receive the value.
- A sign-up flow that stops at a CAPTCHA, a phone check or a payment page and states what the
  person must do.
- Cross-process leases, so several agents can work on different windows at the same time.
- A focus guard that reverts a focus change the person's input does not explain, and pauses
  the agents when the person moves into the agent's area.
- File, background-job and checkpoint tools that also run without Windows dependencies
  (`harness/native_server.py`).

## Requirements

- Windows 10 or 11
- Python 3.11 or 3.12
- Chrome or Edge, for browser tasks
- An MCP client: Claude Code, Codex or Kimi Code

## Installation

```
git clone https://github.com/52kokada/computeruse-windows.git computeruse
cd computeruse
powershell -ExecutionPolicy Bypass -File scripts\install.ps1
.venv\Scripts\python -m harness.doctor
```

Register the server with an MCP client. The server name is `computeruse` in Claude Code and
`computeruse_desk` in Codex.

```
claude mcp add --scope user computeruse -- "<repo>\.venv\Scripts\python.exe" "<repo>\harness\server.py"
codex mcp add computeruse_desk -- "<repo>\.venv\Scripts\python.exe" "<repo>\harness\server.py"
```

See [docs/new-desktop.md](docs/new-desktop.md) for the full procedure, including Kimi Code.

## Usage

An agent starts with `desk()`, which lists every window and what it contains. It reads a window
with `observe` and acts with `act`, or uses the browser operations for web pages.

```
desk()
launch(app="chrome")
browser(op="tabs")
capabilities(query="office")
```

`capabilities()` lists every tool, and `capabilities(query=...)` returns the matching schemas.

## Configuration

Settings are in `config.yaml`.

| Key | Purpose |
|---|---|
| `ai_territory` | Where the agent works: `desktop`, `monitor` or `auto` |
| `ai_territory_lock`, `ai_monitor_ok` | Limits on the territory. Only the person changes these |
| `chrome_profile` | The Chrome profile that the agent's browser windows use |
| `repo_roots` | Folders in which file and code tasks may work |
| `allow_signups`, `allow_signins` | Let sign-up and sign-in presses through the gate |
| `allow_sends`, `allow_email_send`, `allow_deletes`, `allow_payments`, `allow_permissions` | Let that kind of action through without a sign-off |

Every `allow_*` setting is `false` in this repository.

## Security

- A step that sends, deletes, pays or grants a permission stops until the person approves that
  specific action with `python -m harness.gate approve <key>`. No MCP tool can approve
  (`harness/gate.py`).
- Text that the agent reads has hidden and control characters removed, and page text is placed
  between nonce markers. A URL is refused when its path or query carries text copied from an
  earlier read of a different site (`harness/egress.py`).
- A task handed to a worker runs the Claude or Codex command line with its own approval
  prompts off (`harness/headless.py`). The gate is the control on side effects. Read
  `harness/gate.py` before changing an `allow_*` setting.

To report a vulnerability, open a private security advisory on this repository.

## Documentation

- [docs/setup.md](docs/setup.md) and [docs/new-desktop.md](docs/new-desktop.md): installation
- [docs/reference.md](docs/reference.md): tools, dashboard, diagnostics
- [docs/native-tools.md](docs/native-tools.md): file, job and checkpoint tools
- [docs/java-apps.md](docs/java-apps.md): Java apps
- [docs/websites.md](docs/websites.md): site shortcuts
- [docs/multi-pc.md](docs/multi-pc.md): work on another PC
- [docs/probes/](docs/probes/): measurements of input methods and virtual desktops

## Development

```
.venv\Scripts\python -m pytest
```

The default run uses fakes and does not drive the desktop. Tests that drive a real desktop or
browser run only when `COMPUTERUSE_LIVE` is set.
`tests/test_password_vault.py::test_real_test_prefix_export_import_roundtrip` writes test
entries to Windows Credential Manager; deselect it to avoid that.

Known failure: `test_instructions_fit_what_a_client_shows`. The server instruction text is
longer than the limit the test sets.

### Benchmark tasks

`bench/tasks.yaml` defines everyday tasks such as filling a form, editing a workbook, changing
a file and running a git operation. `harness/bench.py` submits each task through the API that
an agent uses, times it, and checks the result from outside the harness: the file on disk, the
workbook read back, or the form that the local fixture site received. Of the 21 fixture pages
in `bench/site/`, 13 are injection attempts.

| Path | Contents |
|---|---|
| `harness/` | The harness. `session.py` is the tool dispatcher, `browser.py` the browser driver, `winapi.py` the Windows layer |
| `tests/` | Test files |
| `bench/` | Benchmark tasks and the fixture site |
| `knowledge/` | App profiles, routines and traces that the harness writes at run time |
| `scripts/` | Install and maintenance scripts |

## Project status

computeruse is in daily use on a small number of Windows PCs and runs on Windows only. Some
apps have no input channel that works without the cursor; `knowledge/apps/` records what works
for each profiled app. A few names (the "First Harness" voice app, the `kokuban` agent board)
refer to other parts of the installation it was written for.

This repository is a public copy of a private working repository. Learned run data is removed,
and people, PCs, addresses and businesses in tests and docs are invented.

## License

[MIT](LICENSE)