guarded-computer-use-mcp

Give your agent hands — without giving away the keys.

A Windows computer-use MCP server where every risky action stops and waits for a real human click. 26 tools: screenshot, mouse, keyboard, UI Automation, OCR, windows, clipboard.

中文文档

Why

Most computer-use tools hand the model a keyboard and hope for the best. That is fine until the model mis-clicks into a chat window, an email, or a payment page — and there is no undo.

This one puts a gate in front of the dangerous part:

The MCP server blocks on that dialog and uses its exit code. Only a real human action can proceed — the model cannot forge one.

Built so a stray keystroke cannot approve anything:

The dialog also shows the actual arguments being executed (automationId=…, name=…), follows your OS language (Chinese/English), beeps, and stays on top. Tick remember this target for this session to stop being asked about the same tool + target until the server restarts.

Exit codes: 0 allow once · 1 deny · 2 timeout · 3 dialog unavailable (falls back to pending_safety_check) · 4 allow + remember for this session.

Flip it off any time when you don't want to be interrupted:

double-click  guard-panel.cmd      # the panel: three independent switches
double-click  toggle-approval.cmd  # quick toggle for the dialog only

The panel writes a marker file per switch, and the server reads them on every call — changes apply instantly, no restart:

They are independent: turning off the deny lists does not silently turn off the dialog.

Related MCP server: desktop-touch-mcp

Three layers of protection

Target resolution uses WindowFromPoint — which window a click actually lands on, not the foreground window:

{ "blocked_by_policy": true, "reason": "target process is on the deny list",
  "detail": { "process": "ApplicationFrameHost", "title": "计算器",
              "matched": "applicationframehost" } }

What is on the lists

Four lists, all substring matches (case-insensitive), all in policy.json:

The split matters: a messaging app is not denied outright — you may want the agent to read or summarise it — but nothing is clicked there without you saying yes.

npm run test:policy checks the lists against 29 samples and fails on false positives (a browser, Notepad, Blender and this repo's own harness must all pass cleanly).

Install

Requirements: Windows 10/11, Node.js ≥ 18.

git clone https://github.com/be1st6666/guarded-computer-use-mcp
cd guarded-computer-use-mcp
npm install

Optional but recommended: PowerShell 7 (better UTF-8 and JSON handling). The server falls back to Windows PowerShell 5.1 automatically.

Optional: OCR needs uv on PATH. Everything else works without it.

Verify it works

npm test              # read-only tools; no side effects, safe to run any time
npm run test:policy   # the deny/approval lists, 29 samples
npm run test:typing   # activate_window + multi-line type_text (launches Notepad)

npm test prints a tick per tool plus its latency. If it lists tools and screenshot returns an image, the server is wired up correctly.

Configure your MCP client

{
  "mcpServers": {
    "computer": {
      "command": "node",
      "args": ["D:/path/to/guarded-computer-use-mcp/server.js"]
    }
  }
}

Add to $DSH_HOME/profiles/web/cordis.patch.yml:

- insert:
    - id: mcp-computer-use
      name: '@deepseek-ai/dsh-mcp-client'
      config:
        serverName: computer
        transport: stdio
        command: C:/Program Files/nodejs/node.exe
        args:
          - D:/path/to/guarded-computer-use-mcp/server.js

Tools appear as mcp__computer__<name>.

Performance

Measured on a 2560×1600 display, median of 5–8 runs:

The ~55 ms capture floor is the GPU→CPU readback of a full frame over GDI.

OCR latency scales with the number of recognised text boxes, not with the capture rectangle: a text-heavy full 2560×1600 screen measured ~6.9 s / 105 boxes, while a 1280×800 region measured ~3.3 s / 45 boxes (~2×). Downscaling the image first does not help — RapidOCR's detector normalises its input to a fixed size internally, so pre-scaling costs accuracy for almost no time — and a larger recognition batch is slower, not faster. So pass a region (x/y/width/height) instead of OCRing the whole screen whenever you do not need all of it.

Token economics

A screenshot costs ~2133 tokens. The alternatives cost far less:

batch also collapses N actions into one model turn, which is the bigger win: a model turn costs 2–10 s of inference, a tool call costs 1–457 ms.

Three-tier targeting

Coordinates fail silently when a window moves. So try in order:

1. find_elements / click_element — UI Automation. Clicks by name via InvokePattern (no mouse movement at all). Either finds the element or says it isn't there.

2. ocr — no control tree, but there is text. RapidOCR (PaddleOCR models on ONNXRuntime, ~14 MB) reads it and returns screen-space boxes, so a recognised word can be clicked directly.

   

   Real output against the Windows Calculator: every digit located with a confidence score. The built-in Windows OCR engine returns zero words for this same region.

3. screenshot + coordinates — last resort for canvas/self-drawn UI.

What the accessibility tree actually covers. It is easy to assume browsers expose nothing and reach for pixels too early. In practice Edge/Chromium does publish the page: a single find_elements on a 4399.com game index returned 22 hyperlinks with exact rects, including ones thousands of pixels offscreen — all clickable by name with no mouse movement.

What it still cannot see:

When it is not, ocr and then plain coordinates are the fallbacks — in that order.

Measured on the same task (click a button):

Tools (26)

Coordinates are physical pixels of the virtual desktop. type_text injects Unicode per character via SendInput, so CJK, quotes and emoji go through verbatim with no escaping anywhere in the path.

Event-driven, not a video stream

Codex-style computer use is not a continuous stream either — it is act → screenshot → decide. Three primitives make that loop cheap:

• screen_hash — 64×40 perceptual fingerprint, 44 ms, 9 tokens

• wait_for_change — blocks until the screen actually changes; no polling from the model, no wasted turns

• batch — [click, wait, screenshot] in one call, one round trip

Architecture

MCP client
   │  stdio, JSON-RPC
   ▼
server.js ──── one base64(UTF-8 JSON) line per request ────▶ host.ps1 (resident)
   ▲                                                              │
   └─────────── one base64(UTF-8 JSON) line per reply ◀──────────┘
                                                                  │
                                                    C# class Dsh (compiled once)
                                                                  │
                                              StretchBlt / SendInput / UIAutomation

Design notes:

• Resident host, compiled once. The C# helper is Add-Typed at startup, so a call costs 1–12 ms instead of 300–500 ms.

• StretchBlt capture + downscale in one GDI call. Capture-then-resample spent 57 ms just on resampling — more than encoding.

• JPEG by default. PNG encode of 1600×1000 costs 19–33 ms; JPEG q88 costs 4–8 ms and is half the size.

• base64 transport. Every line on the wire is ASCII, sidestepping Windows PowerShell 5.1 console-encoding entirely.

Safety model in depth

What it protects against

• The model deciding on its own to touch a password manager, banking or payment window → refused by policy, no override.

• A destructive click (关闭 / Close / Delete / Send) or key chord (alt+f4, ctrl+w, win+*, shift+delete) → approval dialog.

• Runaway loops → max_actions_per_minute.

• "What did it actually do?" → audit.jsonl with the target process per action.

What it does not protect against

• A mis-click in an app that is not on a deny list. If the model clicks the wrong thing in Notepad, nothing stops it.

• It is not a sandbox. The agent runs with your user's privileges on your real desktop. There is no VM. "Control the real machine" and "full isolation" are architecturally mutually exclusive — Codex's sandbox works because it drives a desktop inside a VM, not yours.

• Elevated windows: UIPI blocks input injection into admin windows, and the UAC secure desktop is unreachable. This is a Windows boundary, not a feature.

Risks and disclaimer

What can go wrong

This is not a toy permission. An agent driving this server can:

• send a message, email or payment as you

• delete or overwrite files it was never asked to touch

• read whatever is on your screen, including other people's private data

• change application or system settings

There is no undo. Ctrl+Z does not cover "sent", "paid" or "deleted".

Prompt injection

The agent reads your screen and the documents on it. Anything it reads can carry instructions: a web page, a PDF, an email, a chat message, a code comment. A hostile page can tell the agent to do something you never asked for.

The approval gate catches actions matching the safety patterns and the approval lists. It does not catch a plausible-looking click in an app that is on neither list. Treat everything the agent reads as untrusted input.

Your screen leaves your machine

Screenshots, OCR output, clipboard contents and window titles are sent to whichever model provider your MCP client uses. That is the whole point — the model has to see the screen — but it means:

• everything visible while the agent runs is transmitted off-device

• that includes other people's messages, documents and personal data

• check your provider's data-retention policy before pointing this at anything sensitive

• prefer zoom on a small region over a full screenshot when you can

OCR runs locally and sends nothing itself, but its output is returned to the model, so it is transmitted too.

What the guard does not do

• It is not a sandbox. The agent runs as you, on your desktop, with your sessions and tokens.

• It does not stop a wrong action in an app that is on neither list.

• It does not survive anyone who can also edit policy.json or create the .guard-off marker — those are plain files in this directory.

• The deny lists are substring matches: a speed bump, not a boundary.

You are responsible

You choose what to point this at, which lists to keep enabled, and whether to leave the approval gate on. Run it with the gate on until you have watched it work on your own machine. Do not expose the stdio transport to a network.

Provided under the MIT licence, without warranty of any kind — see LICENSE. The authors are not liable for any loss or damage arising from its use.

Known limits

• Windows only.

• ~55 ms per full screenshot — the GDI readback floor. A DXGI Desktop Duplication backend could cut it, at the cost of a native addon.

• No continuous vision. By design; see the event-driven section.

• OCR costs ~0.5–2.5 s depending on whether the warm worker is alive.

• type_text needs focus — activate_window + click first.

Development

npm test                      # read-only tools, no side effects
npm run test:uia              # accessibility tree + semantic search
npm run test:policy           # deny/approval lists, 29 samples, fails on false positives
npm run test:typing           # activate_window focus + multi-line/tab type_text
npm run bench                 # latency + token table
node test-client.js policy    # policy + audit behaviour end to end
node test-client.js rapid     # Windows OCR vs RapidOCR on the same region
node test-client.js newtools  # launch_app, approval gate, OCR-driven click

docs/make-*.ps1 regenerate the README figures from a live screen, so the screenshots can be kept honest rather than hand-drawn.

host.ps1 and ocr.ps1 must stay pure ASCII: Windows PowerShell 5.1 reads .ps1 as ANSI when there is no BOM, and a single non-ASCII byte can swallow a newline and corrupt the embedded C#. Verify with:

((Get-Content .\host.ps1 -AsByteStream) | Where-Object { $_ -gt 127 }).Count   # must be 0

approval.ps1, approval-toggle.ps1 and guard-panel.ps1 are the exception: they show Chinese to the user, so they are saved with a UTF-8 BOM, which both shells honour.

Contributing

Issues and PRs welcome. Two rules keep this repo reviewable:

1. No third-party automation code. The whole point is that every line that touches your machine can be read in one sitting. host.ps1 is C# + Win32 and nothing else.

2. Measure, don't claim. If you change something for speed, put a number in the PR — npm run bench exists for that.

License

MIT — see LICENSE.