Skip to main content
Glama

computeruse

computeruse is a computer-use harness for Windows. It exposes desktop apps and browsers to AI agents (Claude, Codex, Kimi) as MCP tools, and confines the agent to its own virtual desktop or monitor so that a person can keep using the same PC.

Features

  • Reads windows through UI Automation and acts with messages posted to the window, without moving the person's cursor or using their keyboard or clipboard.

  • A territory for the agent: a virtual desktop, a monitor, or half of a single monitor. The agent can observe any window and can act only inside its territory or on windows that the person hands over.

  • Browser control for Chrome and Edge: tabs, forms and page reads.

  • Drivers for Office documents and for Java (Swing/AWT) apps through the Java Access Bridge.

  • A sign-off gate for actions that send, delete, pay or grant permissions.

  • A password store in Windows Credential Manager. Forms take a {secret:name} placeholder, and the model does not receive the value.

  • A sign-up flow that stops at a CAPTCHA, a phone check or a payment page and states what the person must do.

  • Cross-process leases, so several agents can work on different windows at the same time.

  • A focus guard that reverts a focus change the person's input does not explain, and pauses the agents when the person moves into the agent's area.

  • File, background-job and checkpoint tools that also run without Windows dependencies (harness/native_server.py).

Related MCP server: guarded-computer-use-mcp

Requirements

  • Windows 10 or 11

  • Python 3.11 or 3.12

  • Chrome or Edge, for browser tasks

  • An MCP client: Claude Code, Codex or Kimi Code

Installation

git clone https://github.com/52kokada/computeruse-windows.git computeruse
cd computeruse
powershell -ExecutionPolicy Bypass -File scripts\install.ps1
.venv\Scripts\python -m harness.doctor

Register the server with an MCP client. The server name is computeruse in Claude Code and computeruse_desk in Codex.

claude mcp add --scope user computeruse -- "<repo>\.venv\Scripts\python.exe" "<repo>\harness\server.py"
codex mcp add computeruse_desk -- "<repo>\.venv\Scripts\python.exe" "<repo>\harness\server.py"

See docs/new-desktop.md for the full procedure, including Kimi Code.

Usage

An agent starts with desk(), which lists every window and what it contains. It reads a window with observe and acts with act, or uses the browser operations for web pages.

desk()
launch(app="chrome")
browser(op="tabs")
capabilities(query="office")

capabilities() lists every tool, and capabilities(query=...) returns the matching schemas.

Configuration

Settings are in config.yaml.

Key

Purpose

ai_territory

Where the agent works: desktop, monitor or auto

ai_territory_lock, ai_monitor_ok

Limits on the territory. Only the person changes these

chrome_profile

The Chrome profile that the agent's browser windows use

repo_roots

Folders in which file and code tasks may work

allow_signups, allow_signins

Let sign-up and sign-in presses through the gate

allow_sends, allow_email_send, allow_deletes, allow_payments, allow_permissions

Let that kind of action through without a sign-off

Every allow_* setting is false in this repository.

Security

  • A step that sends, deletes, pays or grants a permission stops until the person approves that specific action with python -m harness.gate approve <key>. No MCP tool can approve (harness/gate.py).

  • Text that the agent reads has hidden and control characters removed, and page text is placed between nonce markers. A URL is refused when its path or query carries text copied from an earlier read of a different site (harness/egress.py).

  • A task handed to a worker runs the Claude or Codex command line with its own approval prompts off (harness/headless.py). The gate is the control on side effects. Read harness/gate.py before changing an allow_* setting.

To report a vulnerability, open a private security advisory on this repository.

Documentation

Development

.venv\Scripts\python -m pytest

The default run uses fakes and does not drive the desktop. Tests that drive a real desktop or browser run only when COMPUTERUSE_LIVE is set. tests/test_password_vault.py::test_real_test_prefix_export_import_roundtrip writes test entries to Windows Credential Manager; deselect it to avoid that.

Known failure: test_instructions_fit_what_a_client_shows. The server instruction text is longer than the limit the test sets.

Benchmark tasks

bench/tasks.yaml defines everyday tasks such as filling a form, editing a workbook, changing a file and running a git operation. harness/bench.py submits each task through the API that an agent uses, times it, and checks the result from outside the harness: the file on disk, the workbook read back, or the form that the local fixture site received. Of the 21 fixture pages in bench/site/, 13 are injection attempts.

Path

Contents

harness/

The harness. session.py is the tool dispatcher, browser.py the browser driver, winapi.py the Windows layer

tests/

Test files

bench/

Benchmark tasks and the fixture site

knowledge/

App profiles, routines and traces that the harness writes at run time

scripts/

Install and maintenance scripts

Project status

computeruse is in daily use on a small number of Windows PCs and runs on Windows only. Some apps have no input channel that works without the cursor; knowledge/apps/ records what works for each profiled app. A few names (the "First Harness" voice app, the kokuban agent board) refer to other parts of the installation it was written for.

This repository is a public copy of a private working repository. Learned run data is removed, and people, PCs, addresses and businesses in tests and docs are invented.

License

MIT

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables agentic control of Windows via MCP tools for screenshots, mouse, keyboard, UI Automation, OCR, windows, and clipboard, while risky actions require real-human approval through a policy-gated dialog with audit logging.
      5
      MIT
    • A
      license
      A
      quality
      C
      maintenance
      Enables MCP-speaking AI agents to operate a desktop and browser through structured UI inspection, mouse/keyboard gestures, tab-scoped browsing, and approved local screen recording with editing.
      3
      1
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables AI agents to see and operate Windows applications through MCP, including windows, screenshots, mouse, keyboard, scrolling, UI Automation, clipboard, and freehand or geometric drawing. Supports supervised creative workflows with native Win32 input, screenshots, allow-listed apps, and an idle safety guard.
      MIT