Computer Use

Introduction

Computer Use is an open-source MCP server — a computer-use agent (CUA) backend — that lets any coding agent use your computer the way a person does. It reads the screen through accessibility trees, clicks and types in the background so your mouse stays yours, shows an agent pointer where it is working, zooms in on small text, and drives tabs in your signed-in Chrome — on macOS, Windows and Linux.

Works with Claude Code, Codex, Cursor and MT Code, or any other MCP client, with any model — no vision model is required for interaction.

Built by Munim Technologies as the Computer Use engine of MT Code, and published here on its own.

Related MCP server: openowl

Table of contents

• Quick start

• Capability matrix

• Benchmarks

• Why it works well

• Works alongside you

• Apps and sites the agent may use

• Tools

• Repository layout

• Environment flags

• Embedding in an app

• Prompting your agent

• Contributing

• Credits and license

Quick start

1. Download the latest binary for your platform from Releases (munim-computer-use-macos-universal.zip, munim-computer-use-windows-x64.zip) or build from source.

2. Put it somewhere on your PATH (/usr/local/bin/munim-computer-use, or %LOCALAPPDATA%\Programs\munim-computer-use\munim-computer-use.exe).

3. macOS only: run munim-computer-use request-permissions once to be prompted for Accessibility and Screen Recording.

4. Register it with your agent:

# Claude Code — fastest: the npm launcher fetches the signed binary on first run.
# Add `--scope user` to register it for every project instead of just this one.
claude mcp add munim-computer-use -- npx -y munim-computer-use
# or point at a downloaded binary
claude mcp add munim-computer-use -- /usr/local/bin/munim-computer-use

# Codex — writes the entry below into ~/.codex/config.toml for you (needs a Codex CLI
# with `codex mcp`; check with `codex mcp --help`)
codex mcp add munim-computer-use -- npx -y munim-computer-use

# Codex — ~/.codex/config.toml, if you would rather edit it yourself
[mcp_servers.munim-computer-use]
command = "npx"
args = ["-y", "munim-computer-use"]

# Cursor — no MCP subcommand in its CLI, so write the config. ~/.cursor/mcp.json applies
# to every project; .cursor/mcp.json in a repo applies to that one.
mkdir -p ~/.cursor && [ -s ~/.cursor/mcp.json ] || echo '{}' > ~/.cursor/mcp.json
jq '.mcpServers["munim-computer-use"] = {"command":"npx","args":["-y","munim-computer-use"]}' \
  ~/.cursor/mcp.json > ~/.cursor/mcp.json.tmp && mv ~/.cursor/mcp.json.tmp ~/.cursor/mcp.json

// Cursor — the entry that produces, in .cursor/mcp.json
{ "mcpServers": { "munim-computer-use": { "command": "npx", "args": ["-y", "munim-computer-use"] } } }

Then ask: "Open Safari, find the cheapest flight to Denver on Tuesday and put it in a note." The agent reads the UI with get_app_state, acts by element id, and verifies with screenshot.

Capability matrix

Munim Computer Use is the highlighted first column; the others are the computer-use servers people reach for. each cell comes from that project's own README or docs in September and October 2026 (sources under Credits and license). ✅ present · ❌ absent or not documented · ⚠️ partial.

Also looked at: mediar-ai/mcp-server-macos-use (macOS, accessibility, real input), deploymenttheory/windows-mcp-server (Windows, UIA Invoke patterns, WaitFor), nuphus-mcp (OCR + bring-your-own vision model, CDP Chrome), computer-control-mcp (PyAutoGUI + OCR), and microsoft/playwright-mcp (browser only). Corrections welcome — open an issue with a link.

The chart counts the table above (n/a cells are left out), so it shares the table's bias: we picked the rows.

Benchmarks

Reading a window is the step an agent repeats most, so every server we could run reads the same file manager window on each system as the folder grows from 100 to 20,000 files: File Explorer on Windows, Finder on macOS and Thunar on Linux. Each one ran its latest release on 4 November 2026. munim-computer-use is the fastest at every folder size on all three, and returns the fewest tokens on all three.

Windows 11 · File Explorer

Windows-MCP 0.8.8 takes about 15 s per Snapshot on this PC, where 0.8.5 took 0.7 s on the same windows the same day (and in our 27 October run), so the slowdown is in that release. At 1,000 files its last read left the File Explorer window out (1.2K tokens, no files). munim lists 16 rows and the others 17: munim's on-screen check leaves out the row at the bottom edge of the list.

macOS 27 · Finder

A MacBook Air (M1) on macOS 27.2. munim reads one window here (window: 0), as the others do; without it, get_app_state lists every Finder window. Cua Driver ran with mcp --direct and open-computer-use with OPEN_COMPUTER_USE_DISABLE_APP_AGENT_PROXY=1, their documented switches for running inside the process that holds the permissions, since their own apps had none on this Mac. Cua Driver's read stops at 99 elements, which leaves out part of the file list; zavora lists rows scrolled out of view too. MacOS-MCP's reads swing between about 0.3 s and 17 s at 5,000 files and take over a minute at 20,000.

Linux · Thunar

Ubuntu in WSL 2 on the same laptop, X11 (Xvfb, 1920×1080) with bspwm and AT-SPI. Thunar's default icon view tells assistive technology nothing about its files, so every server read its details view. Only munim lists exactly the 33 rows in view: Cua Driver's read listed files only at 100 files, zavora's tree lists none of them, and open-computer-use lists every row it has, in or out of view, and timed out at 20,000 files on some calls. munim 0.9.3 read nothing of the 20,000-file folder, because AT-SPI refuses to hand over more than 65,536 children at once; 0.10.0 walks such a list by index.

How it was measured

For each folder size a fresh file manager window was opened in front and every server was started once, called over stdio with its own read tool (text only where the tool has a switch for it), one warm-up call dropped, then 5 timed calls; the whole run was done twice, so each time is the median of 10. Tokens are estimates at 4 characters per token. "Files listed" counts the folder's file names in the 1,000-file read. Terminator was left out: its get_window_tree picks the first explorer process, which is the taskbar, and it takes no pid. The scripts are in a scratch kit, not this repository; ask in an issue if you want them.

Acting

On the same Windows PC: the five clicks of 1 + 2 = on Calculator (clear, 1, +, 2, =), each a separate call by element id, after one read. Median of 3 runs.

Most of a click is the agent pointer flying to the target so you can see where the agent acts. run_steps sends all five in one call.

Every Windows release

The same window read by every Windows release, from the release binaries, in two rounds of all 17 with Windows-MCP in each round, plus 0.8.0 measured against 0.7.3 the same way, on 27 October 2026 (an earlier run than the ones above, so the times differ a little; Windows reads have not changed since 0.8.0, and 0.9.3 and 0.10.0 measured alike above). 0.7.0 is where get_app_state started reading only what is on screen and fetching each element's attributes in one request; 0.8.0 fetches the whole window in one request instead of one per element, with the same output.

node scripts/readme-charts.mjs redraws the charts; the coverage chart is counted from the capability matrix.

Why it works well

• Accessibility first, pixels second. get_app_state returns the app's accessibility tree with stable element ids, so the agent presses the button instead of guessing at a coordinate. It costs a fraction of the tokens of a screenshot and it is what scores highest on OSWorld-style tasks. Screenshots are for verifying and for content the tree cannot describe.

• What is on screen, fast. get_app_state lists what is in view: a list or table gives its visible rows, and anything scrolled away is skipped and counted (offscreen: true lists it too). Attributes are read in one request per element, and on Windows the whole window comes back in a single request. Rows of a list print one to a line. A Finder window reads in about 0.05 s, File Explorer in about 0.12 s and Thunar in about 0.2 s, faster than any other server we measured on each system (see Benchmarks).

• Dialogs first. While a sheet, alert or popover blocks a window, only its controls are listed, with a note saying so, so the agent answers the dialog instead of clicking behind it.

• Electron and Chromium apps (macOS). Slack, VS Code, Discord and other Chromium-based apps expose their tree only to a client that asks; get_app_state asks, and says so while a background app is still building it.

• Text the tree cannot describe. When an app exposes no labelled controls, its window is read with on-device OCR (Apple Vision on macOS, Windows OCR on Windows) and the text is listed with ids that click, hover, right_click and drag accept. The image never leaves the machine. ocr: "always" or "never" overrides it.

• Ids that fail instead of misfiring. Every id is rechecked before an action. If its element is gone, or its role or name changed (a dialog swapped its buttons, a list re-sorted), the action fails with a clear error instead of acting on whatever sits there now.

• Waits as long as the app takes. return_state waits for the app's accessibility notifications to stop instead of a fixed pause: about 230 ms for a field edit (was 350 ms), and long enough for a sheet to finish sliding in.

• Background control. Events are addressed to the target window (SkyLight on macOS, UI Automation patterns and posted window messages on Windows). On macOS the agent never takes your mouse or keyboard; see Works alongside you for the exact guarantee on each platform.

• Pointer overlay, not your pointer. A soft lavender agent pointer shows where the agent is acting. Your cursor is untouched.

• Coordinates that land. Every screenshot and zoom carries its screen origin and pixels-per-point. zoom captures any region at full physical resolution.

• Your browser, your logins. The Chrome extension gives the agent its own labelled tab group in your signed-in Chrome, and leaves your tabs alone unless you point it at one.

• Or the tab you already have open. browser_list_tabs all=true shows every tab in the browser and browser_use_tab takes one over in place — useful when the page is already signed in or mid-flow and re-opening the URL would throw that away. An adopted tab is not moved into the agent's group, not activated and not reloaded; cleanup releases it rather than closing it, and browser_release_tab hands it back early.

• Tabs the agent's clicks open are the agent's too. A target=_blank link or popup opened by an agent click or key press within 10 seconds joins that task's tabs and group, and its cleanup closes it. A tab opened from an agent tab without a recent agent action stays yours.

• Parallel tasks, one extension. Every MCP process gets its own tab group, and one process can run several tasks by passing a stable session_id on its browser calls. A task cannot drive or adopt another task's tabs, and cleanup (including a process exiting) closes only its own. Any number of MCP processes share the one extension: the first owns it and the rest go through it, and if the owner exits another takes over without closing anyone's tabs. Tasks share Chrome's cookies and logins, and desktop apps and the clipboard are not isolated.

• Model-agnostic. No vision model is required for interaction; local models work too.

• Look → act → verify. hover for mouse-over menus, wait for loads, query to find a control by label without reading a whole tree.

• Act and look in one call. Pass return_state: true to any action (click, type_text, set_value, browser_click, browser_navigate, …) and the result carries the app's fresh accessibility tree, or the page's current snapshot. Browser actions report readiness separately: a loaded tab is not proof that delayed SPA UI has rendered. Use wait_for_selector for an expected visible element (see below). state_query narrows the desktop tree the same way query does.

• Several steps, one call. run_steps takes a list like click the field, type, press return and runs it in order, each step with the same checks as a direct call. The whole list is checked before anything runs, the first failing step stops the rest, and the result says what ran and what did not. Put return_state on it to see the app after the last step.

• Pages as text. browser_read returns a page's readable text with its headings, including what is scrolled out of view, from a tab in the background. It comes in chunks you can continue with offset, query keeps only the matching lines under their heading, and include_links lists the links.

• Frames too. A support chat, payment form or sign-in box embedded in a page is its own document, often from another site. browser_snapshot lists its controls after the page's, marked (in frame "…"), and their indices work with browser_click, browser_fill and browser_upload_file; browser_read adds each frame's text under a [frame "…"] line, and wait_for_selector looks inside frames. Your site rules apply to a frame's own site, and browser_request_credentials never fills a field inside a frame.

• See a tab without taking your window. browser_screenshot captures what one of the agent's tabs shows, frames included, while it stays in the background: the tab is not selected, its window is not raised and the screen is not captured, so you keep working in your own tab. browser_select_tab bring_to_front=true is the only browser call that takes your window, and it is a last resort.

• JavaScript dialogs, answered on purpose. When a page opens an alert, confirm or prompt, browser_snapshot shows its type, message and id, and other commands on that tab say a dialog is open instead of timing out. browser_handle_dialog accepts or dismisses it, with text for a prompt, in a background tab without the keyboard. A dialog that replaced the one the agent read is refused, and a dialog never counts as an answer on its own.

• Why a page is broken. browser_page_errors lists the uncaught exceptions and console errors a tab's page has logged since it loaded (as far as Chrome keeps them), and the requests that failed (network errors and HTTP 400 or above) since the agent attached, the latest 50, with URLs stripped of query strings and fragments. Exceptions and console errors are read from Chrome's own record during the call, so the DevTools Runtime domain, which bot checks on signed-in sites look for, is not left on in between. Headers, cookies and bodies are never read, and the tab is not reloaded or shown.

• Passwords stay out of the conversation. browser_request_credentials asks the user to sign in in a small Chrome window that shows the site's real origin. What they type goes straight into the page's fields and is never returned to the model. browser_snapshot never shows a password field's value, and on the desktop, typing into password fields is refused by default.

• Pick up what you are looking at. get_app_state and screenshot take app: "frontmost" for the app in front of the user, so a client can hand the agent "this window" in one step.

Works alongside you

The agent has its own pointer; yours stays yours.

macOS — guaranteed. Every action goes through accessibility (press, set value, select text, show menu, scroll bars) or through events addressed to the target app's process and window. The server never moves your pointer, never posts into the system-wide input stream, and never holds or blocks your input, so you can keep clicking and typing in other apps while the agent works — even in the same app, on another window. Events come from a private source, so a modifier you are holding does not leak into the agent's clicks. The target app may be brought forward when that is the point of the step (activate_app, or handing you a password field), but not on every action.

Command shortcuts sent to a background app run through its menu bar (macOS gives menu key equivalents only to the front app), so cmd+W, cmd+S and the like work without bringing it forward. A minimized window, or a hidden app, is moved for the moment it needs pixels or input (typing, a coordinate click, a screenshot, OCR) onto an invisible display, where it works while nobody sees it; it is minimized or hidden again after 20 idle seconds, when another app's window needs the display, or when the server exits. If you bring the app up yourself meanwhile, its windows come back on screen where they were. Accessibility actions on such a window (press, set value) need none of this. COMPUTER_USE_PARK_WINDOWS=0 turns it off, and those actions then return an error asking for the window to be restored.

The exceptions refuse instead of borrowing your pointer: a click, hover or scroll with no target app (coordinates over the desktop before any get_app_state), and drags that leave the source window (between apps, or onto the desktop). The error says what to pass instead.

Windows — best effort, reported. Element presses (Invoke), set_value, select_text and scrolling through UI Automation's ScrollPattern never touch your pointer, and classic Win32 controls also take clicks, hovers, wheel and drags as posted window messages. Other UI (Chromium, Electron, WPF, UWP) only reacts to real mouse input, so those clicks, hovers and drags move your pointer, and type_text/press_key go to the focused window. Whenever the real pointer was used, the result says via cursor. A coordinate action on an element that another app's window covers is refused rather than clicking that window, and pointer and keyboard actions on a minimized window return an error asking for it to be restored (accessibility actions still work on it).

Linux — pointer actions use it. XTEST input moves the real pointer and goes to the focused window, and results say via cursor. Element actions through AT-SPI (press, set value, insert text, select) do not move it.

Apps and sites the agent may use

You decide which apps and websites the agent may touch. Put a policy.json in the server's support directory (~/Library/Application Support/computer-use on macOS, %LOCALAPPDATA%\munim-computer-use on Windows, ~/.local/share/munim-computer-use on Linux; munim-computer-use identity prints it as supportDir), or point COMPUTER_USE_POLICY at a file anywhere. An app that embeds the server under its own profile reads the file from its own support directory:

{
  "apps": { "Keychain Access": "block", "com.apple.MobileSMS": "ask" },
  "sites": { "bank.example": "block", "mail.google.com": "ask", "*": "allow" },
  "foreground": "allow"
}

• allow, ask or block per app or site. ask shows the user a prompt the first time the agent reaches for it (a system dialog for apps, a small Chrome window for sites), and an approval lasts until that agent task ends. An unanswered prompt counts as no after two minutes.

• Apps match their name or bundle id, ignoring case. The rule applies when the agent reads, captures or activates the app. Actions then target elements from a snapshot that was allowed.

• Sites match a host and all its subdomains, and the most specific pattern wins. They are checked on every browser action against the page the tab is showing at that moment, so following a link into a blocked site does not get around it.

• * sets the default, which is otherwise allow, so {"apps": {"*": "block", "Notes": "allow"}} is an allow-list.

• "foreground" is one rule for the whole screen, not per app. Set it to "block" to keep the agent in the background, so it never takes the screen from you. It covers:

  • activate_app;

  • browser_select_tab with bring_to_front;

  • on Windows, typing into a window that is not in front, and any action that would fall back to your real pointer;

  • on Linux, every pointer action (they all move the real pointer).

  Those calls fail with foreground_required before anything moves, and work that runs in the background goes ahead. "ask" prompts once per agent task. A tool argument cannot override the rule. Remote control from a viewer is exempt, because there you are the one driving.

• Changes apply at once, with no restart. A file that is not valid blocks everything rather than being ignored, so a typo cannot quietly turn a block into an allow.

This is a guard rail for an agent that follows its instructions, not a sandbox. A coordinate click lands wherever it points, and a whole-display screenshot shows every window.

Tools (38)

Every action tool also takes return_state, which returns current state after the action in the same call. run_steps runs a list of them in one call and stops at the first that fails.

Editing a page in a background tab

The agent's tabs stay in the background: the browser tools never select them, raise Chrome's window, or send OS keystrokes. While the agent holds a tab, its DevTools session emulates focus, so Chrome keeps rendering the tab and real input reaches it:

• browser_click presses with trusted mouse events, which frameworks that ignore synthetic clicks (Angular Material buttons, for example) respond to. Page events remain only as a fallback, and the reply says which route ran.

• browser_press_key takes any key plus modifiers (cmd, ctrl, alt, shift): Delete, arrows, Home/End, PageUp/PageDown, F-keys, characters and shortcuts. Editing shortcuts such as select all (key: "a", modifiers: ["cmd"]) mean the same with cmd or ctrl on every platform. Browser shortcuts like close-tab never reach Chrome itself.

• browser_fill replaces a field's whole value by its snapshot index: input, textarea or contenteditable, open shadow roots included. It selects everything, inserts the new text through Chrome's text input (trusted beforeinput/input events), and says whether the field now holds exactly that text.

• browser_upload_file attaches local files to a file input by its snapshot index, or by the index of a label for one. Chrome sets the input's files (DOM.setFileInputFiles) and the page gets its usual input/change events. No file picker opens and no OS input is sent. Snapshots list file inputs as input[file], including ones a page hides behind a styled button, and show how many files each holds. Paths are on the computer running Chrome and the server, and the server checks each is a readable regular file before sending it. Several files need an input with multiple. The tool only stages the files: it never submits the form, and the reply says the site has not received them yet.

• browser_snapshot also lists table and grid rows (row "Ana | Admin | Active"), cells and other elements that are only clickable by handler or pointer cursor, so list rows can be found and clicked. Checkboxes, radios and switches are named by their label, never their value ("on"), and show their state: input[checkbox] "Admin (all permissions)" (unchecked).

• A click never lands on the wrong thing. An element out of view is scrolled into view, and the press goes only to a point where the element itself (or its own label) is on top. When a banner, dialog or another control covers it, nothing is pressed and the call fails naming what is in front. A checkbox click reports what the box is now.

The desktop press_key and type_text go to whichever Chrome window is in front, which may be the user's, so never use them on an agent tab.

Typing alongside the user

The person at the computer can keep typing in their own windows while the agent types elsewhere. Browser tools never touch the OS keyboard. On the desktop, delivering keys or a click to a background app lends that app keyboard focus for the length of the action. On macOS, keys the person presses during that moment are held in a short-lived session event tap and replayed, in order, to their own window once it has focus again (set MUNIM_COMPUTER_USE_HOLD_USER_KEYS=0 to turn this off). Windows holds and replays them the same way. A desktop click by element_id that has to use a position checks it with the app's own hit test first, and fails instead of clicking whatever else is there.

Browser readiness and safe re-observation

browser_click, browser_type, browser_fill, browser_upload_file, browser_press_key, and browser_navigate accept wait_for_selector together with return_state: true. browser_snapshot accepts the same wait without performing an action:

{"tab_id": 42, "index": 0, "return_state": true,
 "wait_for_selector": "[role=dialog] button", "wait_timeout_ms": 3000}

The extension polls for any matching visible element in the page, open shadow roots included (each root is matched on its own, so one selector cannot span a host and its shadow tree), independently of resource loading, including same-document rendering. It shares the snapshot's visibility rules: at least 2×2 CSS pixels; no hidden, aria-hidden, inert or fully transparent ancestor. Self-transparent native form controls remain available for styled upload/checkbox hit targets; transparent containers do not. This is not a guarantee of clickability or lack of occlusion. Frames are searched too (a match there names the frame); closed shadow roots are not, and a timeout says how many frames could not be searched. Choose a selector specific to the new expected UI; an element already present can satisfy it immediately. No arbitrary page script is accepted, and a selector may not test the value attribute: React mirrors a controlled input's value there, password fields included, so a wait could otherwise confirm a password one guessed character at a time.

• wait_timeout_ms: defaults to 8000, accepts integers 0–10000; 0 checks once. Requires wait_for_selector. The budget covers polling, not browser/API latency or user site-approval prompts. Site rules are rechecked before every observation and before the final snapshot.

• readiness: met (visible_selector, … ms): the specified condition was observed, not a guarantee that all network requests, animations, or application work ended.

• readiness: timeout: the condition was not observed within the budget. The action's success remains separate, and the current snapshot is still returned.

• readiness: error: the condition could not be evaluated (for example invalid CSS); the action can still have succeeded. Correct the wait and re-observe.

• Without a selector, return_state retains the short pause and bounded navigation-load wait, but reports not_requested (document_load, … ms) rather than claiming SPA readiness. A navigation-load budget expiry reports timeout. A plain browser_snapshot without a wait remains an immediate observation.

If a returned snapshot looks unchanged, do not repeat a non-idempotent click just because expected UI is absent. Wait/re-observe first:

{"tab_id": 42, "wait_for_selector": "[role=dialog] button", "wait_timeout_ms": 3000}

Send this to browser_snapshot, inspect its readiness and fresh indices, then choose the next action. A timeout or tool error does not roll back a click, submission, typing, or navigation. Use current state to decide whether a retry is safe. Invalid wait types/bounds or an action wait without return_state: true are rejected before acting. null for an optional argument means the same as leaving it out. These options require an updated extension as well as the native server; when an older extension ignores them, the result says so and asks for a reload.

Regression checks: node --test chrome-extension/background.test.mjs covers the wait contract and site-policy guards. The real-browser CI harness scripts/e2e-browser.mjs tests a dialog delayed by 1100 ms without navigation, one click only, plus timeout and read-only recovery. The dependency-free CDP regressions cover both snapshot scoping/pagination and SPA readiness. Run against an isolated Chrome/Chromium profile; no Playwright module is required:

node scripts/snapshot-dom.test.mjs --chrome /path/to/chrome
# Extract the pre-fix source; each suite must fail independently:
git show deb4c7a:chrome-extension/background.js > original-background.js
node scripts/snapshot-dom.test.mjs --chrome /path/to/chrome --source original-background.js --suite snapshot
node scripts/snapshot-dom.test.mjs --chrome /path/to/chrome --source original-background.js --suite readiness

browser_snapshot automatically scopes to the topmost visible dialog when one is open, so covered background controls cannot consume the dialog's budget. A dialog's portalled popups stay in scope: elements its controls point at through aria-controls/aria-owns, open popovers, and controls painted on top of the dialog, such as a select's options rendered at the end of <body>. Hidden and inert subtrees are excluded; reachable offscreen controls remain available. Native modeless dialogs and explicit aria-modal="false" do not scope the page. Native modal dialogs and explicit aria-modal="true" are definitive candidates. Implicit ARIA dialogs are heuristically scoped only when they cover the viewport centre and are topmost there, rather than merely at their own centre. With no hit-test stack, centre coverage is the fallback. Corner chat widgets and bottom cookie banners therefore retain the page's indexed controls. It returns up to limit controls (default and maximum 250), reports scope, offset, limit, and total, and explicitly says when it is truncated and which offset to request next. Continue on the same tab_id and session_id. Indices are global within the selected scope and stay actionable across pages of an unchanged UI; after a UI change or navigation, take a fresh snapshot from offset=0. An action's return_state also starts at offset=0 and uses the same scope and truncation reporting.

Names, argument shapes and descriptions are identical on every platform, and CI enforces it (node scripts/check-tool-parity.mjs); a model that learned them on a Mac needs nothing new on Windows. To change a tool, edit both literals at once with scripts/tool-defs.mjs rather than by hand.

Repository layout

Build from source

On Fedora, install the native development libraries before building the Rust server (in addition to your Rust toolchain):

sudo dnf install gcc pkgconf-pkg-config clang-devel pipewire-devel \
  libdrm-devel mesa-libgbm-devel mesa-libEGL-devel wayland-devel libxcb-devel

These are needed by the screen-capture dependencies even on X11. In particular, a missing libpipewire-0.3.pc during the build means pipewire-devel is missing.

# macOS
cd macos && swift build -c release
# Windows / Linux
cd windows-linux && cargo build --release

Linux notes: element actions work everywhere; coordinate clicks need an X11 or XWayland client, since native Wayland apps do not expose absolute geometry.

Tests

cd windows-linux && cargo test                  # Rust server
node chrome-extension/background.test.mjs       # extension, against a fake Chrome
node scripts/check-tool-parity.mjs              # Swift and Rust tool lists match
# Snapshot and SPA readiness regressions (dependency-free CDP, isolated profile):
node scripts/snapshot-dom.test.mjs --chrome <Chrome for Testing binary>
# The browser tools end to end, in a throwaway Chrome for Testing profile that
# never touches your own Chrome (npx playwright install chromium to get one):
node scripts/e2e-browser.mjs --server <munim-computer-use binary> --chrome <Chrome for Testing binary>

Chrome extension (optional)

1. chrome://extensions → Developer mode → Load unpacked → select chrome-extension/.

2. Register the native messaging host: munim-computer-use install-native-host (for example npx -y munim-computer-use install-native-host), or from a checkout sh chrome-extension/install.sh (macOS/Linux) / powershell -File chrome-extension/install.ps1 (Windows), which find the build and run the same command. Point COMPUTER_USE_PATH at the binary if it is not in the default build location.

If the extension still gets Specified native messaging host not found., or the browser tools say the extension is not connected, run munim-computer-use check-native-host in an ordinary terminal. It looks the host up the way Chrome does (on Windows: registry views and policies; everywhere: the manifest's encoding, name, type and allowed origins), starts it with Chrome's command line, and names the step that would fail. Two Windows causes look fine from the inside. An installer run by an MSIX-packaged app (some agent desktop apps are packaged) writes to a private, virtualised copy of HKCU and %LOCALAPPDATA% that Chrome never reads. One run by a sandbox with its own Windows account writes to that account's HKCU. The check detects both; run install-native-host again from your own terminal. install-native-host reports the same findings under problems.

The standalone server's host is com.munimtech.computer_use.desktop; MT Code, which bundles this server, registers com.munim.mtcode.desktop. The extension connects to every host it knows at once and answers each on its own connection, so MT Code and a standalone server (npx, Claude Code, Cursor, a checkout) can both drive Chrome at the same time, each in its own tab groups.

Sandboxed agents (Windows)

Some agents run their commands in a sandbox: a restricted or low-integrity token, a separate Windows account, or another desktop. A server started there can still list windows and their titles, but other apps' UI Automation refuses it and captures come back black, without any error. When that happens the server says so: get_app_state explains an empty tree, screenshot and zoom flag a capture that is one flat colour, and activate_app checks that the window really came forward. Each names the cause it found (for example "runs at low integrity" or "runs as another account"). Retrying from the same context will not help; configure the agent to start the MCP server in your own, unsandboxed context.

Environment flags

Remote control

Normally this server never touches the pointer: coordinate clicks are routed to a specific window, keystrokes are posted to a specific process, and an action that cannot be targeted is refused rather than taking over the machine. That is what lets an agent work while the user keeps using their computer.

COMPUTER_USE_REMOTE_CONTROL=1 inverts that contract for one process, for the case where a person is watching this machine's screen from another one and is steering it themselves. Then click, right_click, drag, hover and scroll move the real cursor and type_text and press_key go to whatever is focused, the way Chrome Remote Desktop or Screen Sharing behave. scroll also accepts x/y so the wheel acts over the point the viewer scrolled at, and the agent-cursor overlay stays hidden — there is only one pointer now.

A host that runs both an agent and a viewer runs them as two processes, so turning this on for the viewer never takes the pointer away from the user on the agent's behalf.

Embedding in an app

An app can ship this binary inside its own bundle and run it under its own identity, so it never shares a browser bridge, agent-cursor app or native-messaging host with a standalone install on the same machine. MT Code does exactly this. Pass a profile, either as a JSON object or a path to a JSON file, with --profile <json|file> (any position) or COMPUTER_USE_PROFILE:

{
  "name": "example-desktop",
  "envPrefix": "EXAMPLE_DESKTOP_",
  "agentCursorName": "ExampleAgentCursor",
  "agentCursorBundleId": "com.example.agent-cursor",
  "nativeHostNames": ["com.example.desktop"],
  "extensionIds": ["abcdefghijklmnopabcdefghijklmnop"],
  "nativeHostDescription": "Example desktop control bridge"
}

Each of supportDir, bridgeSocket, envPrefix, agentCursorName, agentCursorBundleId and historyDir can also be overridden by COMPUTER_USE_<SNAKE_CASE> (for example COMPUTER_USE_SUPPORT_DIR), which wins over the profile. munim-computer-use identity prints the resolved values.

• Browser bridge. Run munim-computer-use install-native-host with the same profile. It writes a wrapper that relays Chrome into this identity's bridge (replaying the profile), and a host manifest for each name in every Chrome/Chromium profile directory (the registry on Windows). It rewrites nothing that is already current, so an app can call it on every launch.

• Extension. Use the stock extension (it connects to com.munim.mtcode.desktop and com.munimtech.computer_use.desktop), or build a variant with its own host names, tab-group title and key: node scripts/build-extension.mjs --out <dir> --host com.example.desktop --group-title "Example" --key <base64>. It prints the variant's extension id for extensionIds.

• macOS permissions. The MCP server is a bare executable, so Accessibility and Screen Recording are granted to the app that spawns it. Only the agent-cursor overlay has a bundle of its own; ship <agentCursorName>.app (a copy of the binary plus an LSUIElement Info.plist) beside the binary, or it is materialised under supportDir on first use.

Prompting your agent

Look → act → verify. get_app_state for ids, act by id with return_state: true to see the result in the same call, and screenshot when the tree cannot show it. Read pages with browser_read, and let the user type passwords through browser_request_credentials. Use zoom for small text, hover for menus that appear on mouse-over, wait after loads, keyboard shortcuts for stubborn widgets. The system-prompt text MT Code gives its agents lives in CodexDeveloperInstructions.ts and is a good starting point.

Contributing

The server is developed here. MT Code ships it by pinning a release of this repository (native/munim-computer-use.json), so changes reach MT Code with the next release. Issues and pull requests are welcome.

Credits and license

Designed and built by Munim Technologies (Munim, Inc.) for MT Code. On-screen pruning, dialog scoping, notification-based settling, OCR fallback, menu shortcuts and window parking follow arc-cua (MIT). Copyright 2026 Munim, Inc. Licensed under the Apache License 2.0; see LICENSE.

Comparison sources: Codex Computer Use, its docs, its Windows notes and MacStories on how it reads apps · Cua Driver and its tool reference · OpenAI Agents API computer use · Anthropic computer-use demo · CursorTouch/Windows-MCP · CursorTouch/MacOS-MCP · QwenLM/open-computer-use · zavora-ai/computer-use-mcp · mediar-ai/mcp-server-macos-use · deploymenttheory/windows-mcp-server · nuphus-mcp · computer-control-mcp · microsoft/playwright-mcp