ComputerUse

ComputerUse is a local desktop-control agent for Ollama vision models. It captures the screen, asks a selected local model for exactly one structured action, executes that small action through local mouse and keyboard tools, verifies the result with a fresh screenshot, and repeats until the task is done.

It is built for the daily computer-use workflow: observe, plan, act, verify.

Video Preview

Watch the demo: ComputerUse local AI desktop agent

Natural-language task
  -> Tauri + React desktop runner
  -> Python worker over newline-delimited JSON
  -> screenshot + perception
  -> Ollama vision planner
  -> Pydantic action validation
  -> local mouse/keyboard executor
  -> verification screenshot
  -> session state, timing events, and UI updates

Related MCP server: helix-pilot

Highlights

• Polished desktop task runner with model selection, screenshot preview, timeline, status, history, and debug timings.

• CLI path for development, dry runs, model listing, screenshots, and automation.

• Local Ollama integration with known vision-capable models ranked first.

• Strict one-action-at-a-time planner contract backed by Pydantic validation.

• Screenshot capture with mss, plus planner overlays and optional UI Automation element collection.

• Local mouse, keyboard, scroll, drag, wait, screenshot, done, and fail action handlers.

• Pause, resume, stop, dry-run, and max-step controls.

• Stdio MCP server for external agents that want safe observe -> execute -> verify computer-use tools.

• Local-only runtime files for the active session, screenshots, timing logs, and run history.

Execution History & Benchmarks

The current local runtime history has been read from data/computeruse.sqlite3 and logs/debug_timing.jsonl. These numbers are a real desktop-run snapshot from timing records dated 2026-07-05 UTC, not synthetic lab benchmarks.

History Snapshot

Most recorded actions were UI-targeted interactions: click_element was the dominant action, followed by type_text, press, done, hotkey, click_target, move, and double_click.

Timing Snapshot

Derived loop overhead from the same timing log:

session_write_ms is not present in the current timing records, so JSON/session-write overhead is not benchmarked in this snapshot.

Requirements

• Windows desktop session.

• Python 3.10 or newer. Python 3.11+ is recommended.

• Ollama running locally at http://127.0.0.1:11434.

• At least one installed Ollama vision model.

• Node.js and npm for the React frontend.

• Rust and Cargo for the Tauri desktop shell.

Quick Start

cd E:\ComputerUse

py -3.11 -m venv .venv
.\.venv\Scripts\Activate.ps1

python -m pip install --upgrade pip
python -m pip install -e .

npm install --prefix apps\desktop

Start the desktop app:

npm run desktop:dev

The Tauri app starts the Python worker with:

python -m computeruse.worker

For the smoothest local setup, launch the desktop app from a shell where the intended Python environment is already active.

Desktop App

The first screen is the task runner, not a landing page.

Run a task like:

Open Chrome, go to YouTube, search for TWICE, and open the first video.

ComputerUse will keep taking one small action at a time until the model returns done, returns fail, the task is cancelled, or the max-step limit is reached.

CLI

List installed Ollama models:

computeruse models

Capture the current screen:

computeruse screenshot

Run a task:

computeruse run "Open Chrome, go to YouTube, search for TWICE, and open the first video" --model llava:latest

Run without executing mouse or keyboard events:

computeruse run "Open Chrome and go to YouTube" --model llava:latest --dry-run --debug-timing

Start the MCP server:

computeruse mcp

Equivalent module entrypoint:

python -m computeruse.mcp_server

Ollama Models

ComputerUse queries Ollama through the local HTTP API and displays every installed model. Known vision-capable families are ranked first when present:

• LLaVA

• BakLLaVA

• Moondream

• MiniCPM-V

• Qwen VL and Qwen2.5-VL

• Gemma vision-capable variants

Example model install:

ollama pull llava:latest

Action Contract

The model may return exactly one JSON object per turn. No prose, no Markdown, no multiple actions.

{
  "thought": "Use the address bar to navigate directly.",
  "action": "hotkey",
  "args": {
    "keys": ["ctrl", "l"]
  },
  "done": false,
  "confidence": 0.92
}

Supported action names:

screenshot
click
double_click
right_click
move
scroll
drag
click_element
double_click_element
move_element
click_target
move_target
type_text
press
hotkey
wait
done
fail

Coordinate actions use absolute screenshot-pixel coordinates from the latest capture. Element and target actions use the most recent UI Automation/perception data where available.

Use click_element for normal controls: buttons, links, tabs, menus, text fields, checkboxes, browser controls, web results, and app navigation. Use double_click_element only for desktop-style items that conventionally require double-click to open: desktop shortcuts/icons, files, folders, Explorer rows, and open/save dialog file rows. Do not double-click web links, buttons, tabs, YouTube thumbnails, checkboxes, text fields, or menu commands. scroll accepts clicks; negative values scroll down and positive values scroll up. drag performs one atomic click-hold-drag-release with start_x, start_y, end_x, end_y, and optional duration_ms.

Worker Protocol

The GUI talks to the Python worker with newline-delimited JSON over a managed process. The protocol is intentionally small and explicit.

GUI commands:

{"type":"list_models"}
{"type":"start_task","task":"Open Chrome and go to YouTube","model":"llava:latest","dry_run":false,"max_steps":50}
{"type":"pause"}
{"type":"resume"}
{"type":"stop"}
{"type":"take_screenshot"}
{"type":"list_history","limit":50}
{"type":"get_history_session","session_id":"..."}

Worker events:

{"type":"models","models":[{"name":"llava:latest","vision":true}]}
{"type":"session_started","session_id":"..."}
{"type":"screenshot","path":"E:\\ComputerUse\\screenshots\\current.png","width":1920,"height":1080}
{"type":"step_started","step_index":3}
{"type":"model_action","action":{"action":"click","args":{"x":500,"y":300},"confidence":0.82}}
{"type":"tool_result","ok":true,"message":"clicked"}
{"type":"timing","step_index":3,"capture_ms":42,"encode_ms":65,"ollama_ms":1840,"execute_ms":7,"session_write_ms":3}
{"type":"session_done","summary":"The requested page is open."}
{"type":"session_failed","reason":"The browser did not load after repeated attempts."}

MCP Server

ComputerUse includes a stdio MCP server so other agents can use local screen observation and one-step execution safely.

Generic MCP configuration:

{
  "mcpServers": {
    "computeruse": {
      "command": "python",
      "args": ["-m", "computeruse.mcp_server"],
      "cwd": "E:\\ComputerUse"
    }
  }
}

If the MCP client does not inherit your activated shell, point directly at the virtual environment:

{
  "mcpServers": {
    "computeruse": {
      "command": "E:\\ComputerUse\\.venv\\Scripts\\python.exe",
      "args": ["-m", "computeruse.mcp_server"],
      "cwd": "E:\\ComputerUse"
    }
  }
}

Codex TOML example:

[mcp_servers.computeruse]
command = 'E:\ComputerUse\.venv\Scripts\python.exe'
args = ['-m', 'computeruse.mcp_server']
cwd = 'E:\ComputerUse'
startup_timeout_sec = 120

Recommended MCP workflow:

1. Call computeruse_help.

2. Call computeruse_start_session with the user task.

3. Call computeruse_observe.

4. Call computeruse_execute_step with exactly one validated action.

5. Inspect the verification observation.

6. Repeat observe/execute until complete.

7. Call computeruse_finish_session.

Runtime Files

These files are local runtime state and are ignored by Git:

The runtime does not need cloud storage. Keep these artifacts private unless you deliberately redact and share them.

Safety Model

ComputerUse is designed around narrow, validated actions:

• The planner cannot run shell commands or arbitrary Python.

• The runtime validates every model action before execution.

• Dry-run mode validates model actions without moving the mouse or typing.

• Pause and stop are available from the desktop UI.

• Passwords, payment details, tokens, destructive changes, purchases, posts, and security-setting changes require explicit user intent before they should be executed.

• Web pages are treated as untrusted input.

Project Layout

apps/
  desktop/                 Tauri 2 + React + TypeScript app
computeruse/
  agent/                   loop, prompts, Ollama client, session and history logic
  schemas/                 Pydantic action and session models
  tools/                   screenshots, screen metadata, mouse, keyboard, windows, executor
  cli.py                   Typer CLI
  worker.py                newline-delimited JSON worker for the GUI
  mcp_server.py            stdio MCP server
data/                      local SQLite history, ignored by Git
logs/                      debug timing logs, ignored by Git
screenshots/               current/planner captures, ignored by Git
sessions/                  active session JSON, ignored by Git

Development

Frontend typecheck:

npm run desktop:typecheck

Frontend/Tauri build:

npm run desktop:build

Python package install in editable mode:

python -m pip install -e .

Troubleshooting

Status

ComputerUse is an MVP local automation tool. Treat real desktop control as powerful and potentially disruptive: start with dry runs, keep tasks specific, and verify the screen after each action.