lowlevel-computer-use-mcp

Real GUI apps. Zero desktop clutter. Give an MCP agent a desktop of its own.

Explore the project site · Jump to quick start

The headline feature is headless desktop automation: run full native GUI applications on an invisible Windows desktop or Linux Xvfb display, then let an agent capture, click, and type without stealing focus or disturbing the desktop you are using.

create isolated desktop → launch real GUI app → automate off-screen → capture & verify

When a login or other human-only step is unavoidable, the agent can temporarily reveal that desktop. A non-dismissible top banner tells the user exactly what to do, and an EMERGENCY EXIT button immediately returns to the normal desktop.

Why headless-with-GUI?

• 🫥 Invisible by default — apps retain a real GUI while staying off your visible desktop

• 🎯 No focus stealing — background mouse, keyboard, and control targeting keep your workflow uninterrupted

• 📸 Still observable — capture individual windows even while they are occluded, unfocused, or off-screen

• 🧑‍💻 Safe human handoff — reveal the desktop only when needed, with persistent instructions and emergency return

• 🪟 Native on Windows — isolated Win32 desktops via CreateDesktop and SwitchDesktop

• 🐧 Native on Linux — virtual GUI displays backed by Xvfb

Related MCP server: computer-use-windows

Everything else agents need

This low-level computer-use MCP server works with Codex, Claude Code, Claude Desktop, and other MCP clients. Headless desktops are the centerpiece; the same server also exposes the primitives needed to automate and verify them end to end:

• 🫥 Headless GUI — run real GUI apps on an off-screen desktop; show them only when a human login is needed

• 🎯 Background / unfocused targeting — drive a specific window via Win32 messages without focusing it

• 📸 Screenshots — all monitors, one monitor, a region, or one window via PrintWindow

• 🖱️ Mouse — move, click, double/right/middle click, drag, scroll, cursor position

• ⌨️ Keyboard — type text, press hotkey combinations (Ctrl+C, Alt+Tab, …)

• 🖥️ Shell commands — run arbitrary system commands and capture output

• 🪟 Windows — list, move, resize, focus, minimize, maximize, restore, close, show/hide

• ⚙️ Processes — list running processes; kill by PID or name

• ✂️ Cropping — crop any saved image to a sub-region

• 🎥 Screen recording — record a monitor or region to mp4 in the background

• 🛡️ Run-as-admin — per-command UAC elevation or whole-server elevation

• 🚀 Silent auto-start — pythonw.exe starts the local HTTP server with no terminal window; elevation is opt-in

• 🟢 AutoHotkey add-in — run AHK scripts; ControlSend/ControlClick for rock-solid background input

• 🐧 Cross-platform — the same tools work natively on Linux (X11 via xdotool/wmctrl), with Xvfb virtual displays for headless-with-GUI

• 🌀 Ephemeral WSL — on a Windows host, spin up a throwaway Linux distro on demand, run commands, tear it down

• 🧩 GUI installer — one window that installs everything automatically

• 💸 Cheap Version — a no-MCP command-line fallback that runs any tool directly from CLI args, for when MCP connections keep failing

⚠️ This server performs real, unsandboxed actions on the host machine — clicking, typing, killing processes and running shell/elevated commands with your user's privileges. Only register it in environments where that is acceptable.

Tools

Every tool returns a JSON string {"ok": true, ...} on success or {"ok": false, "error": "..."} on failure.

Quick start (GUI installer — fully automatic)

The easiest path. It installs uv if missing, runs uv sync, and registers the server with both Claude Code and Codex automatically on launch.

Clone the repo:

git clone https://github.com/codingmachineedge/lowlevel-computer-use-mcp.git

Enter it:

cd lowlevel-computer-use-mcp

Sync once, then launch the GUI-subsystem installer directly. This avoids creating a terminal window for the installer itself:

uv sync
.\.venv\Scripts\lowlevel-computer-use-mcp-installer.exe

Then restart Claude Code / Codex so they spawn the server.

Manual install

Install dependencies and start the stdio server:

uv run lowlevel-computer-use-mcp

Or with pip — install in editable mode:

pip install -e .

Then run it:

lowlevel-computer-use-mcp

Captures (screenshots, recordings) are written to ~/lowlevel-computer-use-captures by default. Override with the LOWLEVEL_CU_CAPTURE_DIR environment variable.

Registering with clients

Replace the path below with wherever you cloned this repo.

Claude Code

Register at user scope (one line):

claude mcp add lowlevel-computer-use --scope user -- "C:\path\to\lowlevel-computer-use-mcp\.venv\Scripts\pythonw.exe" -m lowlevel_computer_use_mcp.server

Or add this to ~/.claude.json under mcpServers:

{
  "mcpServers": {
    "lowlevel-computer-use": {
      "command": "uv",
      "args": ["run", "--directory", "C:\\path\\to\\lowlevel-computer-use-mcp", "lowlevel-computer-use-mcp"]
    }
  }
}

Codex (OpenAI Codex CLI)

Add this block to ~/.codex/config.toml:

[mcp_servers.lowlevel-computer-use]
command = "uv"
args = ["run", "--directory", "C:\\path\\to\\lowlevel-computer-use-mcp", "lowlevel-computer-use-mcp"]
startup_timeout_sec = 60

YOLO mode (auto-approve, no permission prompts)

The GUI installer enables this by default. To do it manually for Claude Code, add a wildcard allow rule for this server's tools to ~/.claude/settings.json:

{ "permissions": { "allow": ["mcp__lowlevel-computer-use__*"] } }

For Codex, set the global approval policy in ~/.codex/config.toml:

approval_policy = "never"

⚠️ YOLO means every tool runs without asking — including destructive ones (kill_process, run_command, run_command_as_admin, wsl_destroy). Only enable it if you trust the agents driving this server.

Cheap Version (no-MCP fallback)

If MCP connections keep failing, you don't need MCP at all. The Cheap Version is a command-line fallback that runs the exact same tool functions in-process and prints the JSON result — no client, transport, or server connection involved.

List every available tool:

uv run lowlevel-computer-use-cheap --list

Take a screenshot:

uv run lowlevel-computer-use-cheap screenshot --monitor 1

Double-click at a point:

uv run lowlevel-computer-use-cheap mouse_click --x 960 --y 540 --clicks 2

Press a hotkey (values are parsed as JSON, so lists work):

uv run lowlevel-computer-use-cheap press_keys --keys '["ctrl","s"]'

Run a shell command:

uv run lowlevel-computer-use-cheap run_command --command "ipconfig /all"

Pass a whole argument object as JSON:

uv run lowlevel-computer-use-cheap screenshot --json '{"window_title":"Notepad"}'

The same thing is also available as a subcommand of the main entry point:

uv run lowlevel-computer-use-mcp cheap get_screen_size

Every parameter from the Tool Reference is accepted as --<param> <value> (or via --json). Output is the identical {"ok": true, ...} JSON the MCP tools return.

Agent skill — every feature documented

A companion skill (skills/lowlevel-computer-use/) teaches agents the entire toolset: a top-level SKILL.md plus reference/TOOLS.md (exhaustive per-tool parameters), reference/WORKFLOWS.md (end-to-end recipes) and reference/PLATFORMS.md (Windows/Linux specifics & gotchas). Install it for Claude Code by copying it into your skills directory:

cp -r skills/lowlevel-computer-use ~/.claude/skills/

The MCP server also ships condensed instructions inline, so any MCP client gets a full feature overview on connect.

Background / unfocused window targeting

Headless execution is the default operating model. mouse_click, type_text and screenshot accept hwnd or window_title. When set, input is delivered to that exact window via Win32 messages without focusing or foregrounding it, and screenshot uses PrintWindow so the window is captured even if it's behind others, minimized, or on an off-screen desktop.

Typical flow:

1. Find the window:

list_windows { "title_filter": "Notepad" }

2. Find the control to target:

list_child_windows { "window_title": "Notepad" }

3. Set its text in the background (most reliable for edit controls):

win_set_control_text { "hwnd": 23924320, "text": "typed without focus" }

4. Or click it in the background (x/y are client coords of the window):

mouse_click { "window_title": "Notepad", "x": 200, "y": 120 }

5. See the result without bringing it forward:

screenshot { "window_title": "Notepad" }

Caveat: message-based input is ignored by some apps (raw input / DirectInput / physical-key-state checks). Try AutoHotkey ahk_control_send; do not silently fall back to foreground input while the user is active.

Foreground mouse, typing, hotkeys, window activation, and interactive desktop switches are focus-protected. They return focus_protected: true unless the call includes confirm_focus_disruption: true after the user explicitly requests a visible handoff.

Headless-but-with-GUI mode

Run a real GUI app on an off-screen Win32 desktop so it never touches your visible desktop, then automate and screenshot it via the background tools.

Create the off-screen desktop:

create_headless_desktop { "name": "work" }

Launch an app onto it:

launch_on_headless_desktop { "name": "work", "command": "notepad.exe" }

List its windows (to get handles):

list_headless_windows { "name": "work" }

Capture a window on it (works even though it's off-screen):

screenshot { "hwnd": 2495156 }

Showing it for an interactive login, then hiding again

Some steps (sign-in) need a human. Temporarily switch the live screen to the off-screen desktop:

show_headless_desktop {
  "name": "work",
  "instruction": "Sign in to the app, then tell the agent you are done.",
  "confirm_focus_disruption": true
}

The interactive desktop has a topmost banner that cannot be closed. It displays the instruction and includes an EMERGENCY EXIT button that immediately returns the user to the normal desktop. After the user finishes, switch back normally:

hide_headless_desktop { "name": "work" }

For an ordinary hidden window on the normal desktop, use show_window / hide_window instead:

show_window { "window_title": "My App", "confirm_focus_disruption": true }

hide_window { "window_title": "My App" }

AutoHotkey add-in

Optional but powerful. AHK's ControlSend/ControlClick drive background windows very reliably, and run_ahk is a full scripting escape hatch.

Install AutoHotkey (one line):

winget install -e --id AutoHotkey.AutoHotkey

Check the server can find it:

ahk_status {}

Send text to a background window by HWND:

ahk_control_send { "text": "hello", "window": "ahk_id 0x1A2B3C" }

Run an arbitrary AHK script (must call ExitApp):

run_ahk { "code": "ControlSendText \"hi\", , \"ahk_exe notepad.exe\"\nExitApp" }

Point the server at a specific AHK exe by setting LOWLEVEL_CU_AHK to its path.

Macros — save repeated sequences as Skills

When you run a multi-step UI sequence the user is likely to repeat, don't leave it as ad-hoc tool calls — capture it as a reusable macro Skill. The server tells agents to do this automatically; see macros/MACRO_SKILL_TEMPLATE.md for the template and rules (resolve handles at run time, prefer background tools, parameterize the variable parts, verify with a screenshot).

Linux (native X11)

The mouse, keyboard, screenshot, process, recording, window management and background/unfocused targeting tools all work natively on Linux. Window control, background input and per-window capture use X11 CLI tools; on Linux hwnd is an X11 window id.

Install the X11 helpers (Debian/Ubuntu):

sudo apt install xdotool wmctrl x11-utils imagemagick xvfb

Check what the server can see:

linux_status {}

Background-type into a window without focusing it (X11):

type_text { "window_title": "Editor", "text": "typed in the background" }

Caveat: X11 background typing uses XSendEvent; most apps accept it, but a few (notably xterm with its default allowSendEvents: false) ignore synthetic events. For those, use Xvfb or report the limitation; do not steal the user's focus.

Headless-with-GUI on Linux (Xvfb)

Start a virtual display:

create_virtual_display { "display": 99, "width": 1280, "height": 800 }

Launch a GUI app onto it:

launch_on_virtual_display { "display": 99, "command": "xterm -e bash" }

List its windows:

list_virtual_display_windows { "display": 99 }

Drive a window on that display (note the display field routes input there):

type_text { "hwnd": 2097164, "display": 99, "text": "hello from headless" }

Capture the whole virtual display:

screenshot_virtual_display { "display": 99 }

Stop it when done:

stop_virtual_display { "display": 99 }

Ephemeral WSL (Linux on a Windows host)

On Windows, spin up a throwaway Linux distro on demand to run Linux software, then tear it down. By default a tiny Alpine minirootfs is downloaded and imported in seconds — your existing distros are untouched.

Check WSL is available:

wsl_status {}

Provision a throwaway distro (downloads latest Alpine minirootfs):

wsl_create_temp {}

Run a command in it (use the name returned above):

wsl_run { "distro": "llcu-tmp-1782754365-53b8", "command": "apk add --no-cache curl && curl --version" }

Tear it down (irreversible — deletes the distro):

wsl_destroy { "name": "llcu-tmp-1782754365-53b8" }

You can also clone an existing distro instead of downloading:

wsl_create_temp { "clone_from": "Ubuntu-24.04" }

Run-as-admin mode

Per-command elevation — call run_command_as_admin (UAC prompt unless already elevated):

run_command_as_admin { "command": "net session" }

Or run the whole server elevated (intended for HTTP mode):

uv run lowlevel-computer-use-mcp --http --admin

Check elevation:

is_admin {}

Console-free startup

Install the default per-user Startup launcher. It uses wscript.exe plus pythonw.exe, opens no terminal window, binds HTTP only to localhost, and needs no UAC prompt:

uv run lowlevel-computer-use-mcp install-startup

An elevated Scheduled Task is an explicit opt-in and may show UAC during setup:

uv run lowlevel-computer-use-mcp install-startup --admin-task

Use a custom port:

uv run lowlevel-computer-use-mcp install-startup --port 9000

Check the task status:

uv run lowlevel-computer-use-mcp startup-status

Remove it:

uv run lowlevel-computer-use-mcp uninstall-startup

Once the boot service runs in HTTP mode, point a client at it as a remote MCP server:

claude mcp add --transport http lowlevel-computer-use-boot http://127.0.0.1:8765/mcp

The startup process owns no top-level window. Tool-spawned Windows child processes also use CREATE_NO_WINDOW plus SW_HIDE. See runtime safety.

Local regex builder

The bundled Python re builder supports guided literals, character classes, anchors, groups, alternation and quantifiers, plus raw patterns, flags, sample text, syntax feedback, captures, copy/export, and plain-text search mode. It evaluates in a console-free subprocess with strict size, time, and match limits.

uv run lowlevel-computer-use-regex-builder --start --literal "ID:" --char-class "0-9" --quantifier "+" --end --sample "ID:42"

uv run lowlevel-computer-use-regex-builder --pattern "^(?P<word>\\w+)$" --flags m --sample "HongKong`nToronto"

Language settings (en, yue, bilingual) and independent English/Cantonese funny levels (1..5) persist only when explicitly changed. Patterns and samples are never persisted or transmitted. See regex builder documentation.

Requirements

• Python 3.10+

• uv (recommended; the GUI installer can bootstrap it)

• Windows: mouse/keyboard via pyautogui; windows via pygetwindow; background input + capture + headless desktop via ctypes/Win32 (winio.py); screenshots via mss; recording via imageio + bundled ffmpeg. Optional: AutoHotkey, WSL.

• Linux: mouse/keyboard via pyautogui (X11); window mgmt, background input and per-window capture via xdotool/wmctrl/x11-utils/ImageMagick (linuxio.py); headless-with-GUI via Xvfb; screenshots/recording via mss. Install the X11 helpers with your package manager (see the Linux section). X11 (or XWayland) session.

Safety notes

• run_command, run_command_as_admin, kill_process, run_ahk and window_action(close) are marked destructive.

• pyautogui's fail-safe is disabled so automation isn't interrupted by the cursor reaching a screen corner; be deliberate with coordinates.

• The server has no authentication of its own — it trusts the MCP client that spawns it.

License

MIT