Deskpass

Why not screenshot-and-click computer use?

The usual way to let a model use a desktop is to send it a screenshot and let it click at pixel coordinates with the real mouse and keyboard. It works on anything that is drawn on screen, and Deskpass keeps it for exactly that. What it cannot do is tell the model what a control is, or leave the desktop to you while the model works.

The screenshot is still there: computer_screenshot and computer_zoom cover what the tree does not, such as web content and canvases.

Related MCP server: Open Computer Use

Install

You need Windows 10 (2004 or later) or Windows 11, and Node.js 22 or newer. No administrator rights and no build tools.

1. Download deskpass-<version>.zip from GitHub Releases.

2. Check it against the SHA-256 in the release notes:

   Get-FileHash .\deskpass-0.2.0.zip -Algorithm SHA256

   Releases are built by the Release workflow from the tagged source, with a signed build provenance. With the GitHub CLI you can check that the zip is the one it built: gh attestation verify deskpass-0.2.0.zip --repo norton77930/deskpass.

3. Extract the zip, open PowerShell in the extracted folder and run:

   powershell -ExecutionPolicy Bypass -File .\install.ps1

The installer copies Deskpass to %LOCALAPPDATA%\Programs\Deskpass, registers it in Claude Code when claude is on PATH, creates the Start menu shortcut "Deskpass Settings", and prints the MCP server definition for your machine and copies it to the clipboard, for any other client (see MCP client setup). Then restart your AI client.

SmartScreen: the helper and the scripts are not code-signed. install.ps1 removes the downloaded-from-the-internet mark from the installed files, so SmartScreen usually stays quiet; if it does warn, check that the zip matches the published SHA-256, then choose "More info" and "Run anyway". If your antivirus or company policy blocks deskpass-helper.exe, ask your IT department rather than turning protection off.

The interface (approval dialog, corner notice, settings page and installer) follows the Windows display language: Traditional Chinese on zh-TW, zh-HK, zh-MO or any zh-Hant language, English otherwise. Set DESKPASS_LANG to en or zh-TW to choose.

To uninstall:

powershell -ExecutionPolicy Bypass -File "$env:LOCALAPPDATA\Programs\Deskpass\uninstall.ps1"

Settings and logs in %LOCALAPPDATA%\deskpass are kept unless you add -RemoveData.

How it works

Deskpass has two processes. A Node.js MCP server holds every policy decision: approval, risk tiers, the single-operator lock, the stop state, screenshot scaling and the action log. A Rust helper does the Win32 work it is told to do: window capture, UI Automation, input, the border and corner notice, the approval dialog and the stop keys.

• Per-app approval. The first time the AI wants an app, you are asked. "Allow once" lasts for the session; "Always allow" is offered only for signed apps and is kept in a signed grants file. Terminals, the registry editor, Task Manager and UAC prompts are blocked; browsers, mail, the file manager and password managers are asked every session.

• UI tree first. computer_get_ui_tree lists a window's controls with numbers; computer_click with element_index, computer_element_action and computer_set_value act on them. Screenshots are scaled to the model's image budget and coordinates are mapped back. To read less of a large app, the AI can pass max_depth (1 to 30 levels; children below are counted as omitted) and/or interactive_only (only numbered elements that can do more than be scrolled into view, and the lines above them), alone or with query. To focus on one pane, root_element_index (with an optional tree_id, default the window's latest tree) reads only the subtree under that element, which counts as level 0 for max_depth; the window's popups are left out. Such a tree begins with a line saying it is not the whole tree (and, for a subtree, which element it is the subtree of); its numbers work like those of any other tree.

• Reading text. computer_read_text reads the text of one element of a UI tree instead of the AI reading it off a screenshot: the text you have selected in it (scope: selection) or all of its text (scope: document, the default), at most max_chars characters (default and most 20,000; longer text is cut there and the answer says more: true, and the AI reads on from there with offset, piece by piece, until more is false). It is approved like a screenshot and takes no input: it does not touch your clipboard, the foreground window, the mouse pointer or your selection. A password field is refused before anything is read, an element with no text to read says so, and an app that does not answer within 3 seconds gives no text at all (each piece has its own 3 seconds, and a later piece of a very long document takes longer than an earlier one, because the text is read from its start). The text comes back in a block of its own, marked as app content and not as your instructions. The action log records the app, the scope and how many characters were read, never the text.

• Waiting for a window or an element. computer_wait_for waits until a window with some text in its title appears, or until a window closes, or until an element whose name or value holds some text appears or goes away (a "Loading" label, say), for up to 30 seconds. Waiting for a window only reads the window list, so it needs no approval. Waiting for an element is approved like reading the UI tree: the app must be allowed and Deskpass takes its lock. Either way it sends no input and does not touch the foreground, and the UI tree the AI already holds stays valid while it waits. If the time runs out it says so, and nothing else has happened.

• Background delivery. Element actions, and scrolling, run through UI Automation without bringing the window forward or moving your mouse pointer. Before each one Deskpass waits until your keyboard has been idle for 0.3 seconds (at most 3 seconds); if the app takes the foreground anyway, it is handed back to your window right away.

• Stop keys. A physical Ctrl+Alt+Esc stops the AI at any time; a single Esc stops it while it works in the foreground. Keys the AI sends itself never count. After a stop, Deskpass asks you before the AI may continue.

• Consent for keys that reach beyond the app. Approving an app, even with "Always allow", does not cover key combinations that act outside it: anything with the Win key, alt+f4, ctrl+alt+delete, alt+tab, ctrl+alt+tab, alt+escape, ctrl+escape, ctrl+shift+escape, ctrl+alt+escape, alt+f8 and alt+f11, also with more keys held. computer_press_key sends them only after you allow that combination for that app in Deskpass's own Windows dialog. A click or a drag holding the Win key needs the same consent, and shares the answer. The answer lasts until the MCP server restarts and is never written to disk, and each question and answer is logged with the app and the combination.

• Logs without content. The log records each action and the app it was on, never typed text, clipboard contents or screenshots.

Design notes, every tool, and the technical limits are in docs/development.md. The terms are defined in CONTEXT.md and the decisions in docs/adr/.

Supported

• Windows 10 (2004 or later) and Windows 11, x64. ARM64 is untested. There is no macOS or Linux version.

• Tested with Claude Code.

• CodeBuddy, Cursor, Claude Desktop and Codex CLI: setup is documented below, and the protocol was verified with a generic stdio MCP client that has no elicitation, but Deskpass has not been tested inside those clients.

• Any other stdio MCP client should work the same way; it is untested.

MCP client setup

Deskpass is a stdio MCP server: the command is node and the one argument is the absolute path of the installed dist\index.js. install.ps1 prints the exact definition for your machine and copies it to the clipboard; paste that. The examples use the default location, %LOCALAPPDATA%\Programs\Deskpass\dist\index.js; replace <you> with your Windows user name. In JSON and TOML every backslash is written twice.

{
  "mcpServers": {
    "deskpass": {
      "command": "node",
      "args": ["C:\\Users\\<you>\\AppData\\Local\\Programs\\Deskpass\\dist\\index.js"]
    }
  }
}

CodeBuddy:

{
  "mcpServers": {
    "deskpass": {
      "type": "stdio",
      "command": "node",
      "args": ["C:\\Users\\<you>\\AppData\\Local\\Programs\\Deskpass\\dist\\index.js"]
    }
  }
}

Codex CLI (config.toml):

[mcp_servers.deskpass]
command = "node"
args = ["C:\\Users\\<you>\\AppData\\Local\\Programs\\Deskpass\\dist\\index.js"]

Any other stdio MCP client works the same way: command node, one argument (the absolute path of dist\index.js), and any server name you like. After editing a client's configuration, restart the client.

Known limitation: a client that does not support MCP elicitation gets the Windows approval dialog instead of an in-client prompt. If that client's tool-call timeout is shorter than the time you take to answer the dialog, the call may fail on the client side even though you approved. In that case approve the dialog, then ask the agent to retry the action; the approval is remembered according to your settings.

The full user guide in Traditional Chinese (first approval, the settings page, background operation, emergency stop, SmartScreen, uninstalling) is docs/user-guide.zh-TW.md.

Evidence

Reproducible numbers only; no speed or success-rate benchmarks.

• npm tests: 711 tests, 707 pass and 4 skipped (the end-to-end tests that open real windows skip without a desktop), run as CI runs them with DESKPASS_NO_DESKTOP=1.

• Rust unit tests: 109 pass (cargo test --bins in helper/).

• Rust desktop tests: the integration tests in helper/tests drive real windows, so CI only checks that they compile; they run on a real desktop.

• CI: every push to master and every pull request runs the TypeScript build, the Rust unit tests and the npm tests on windows-latest (workflow).

Known limitations

• Windows 10 (2004 or later) and Windows 11 only.

• Windows running as administrator cannot be operated, and Deskpass never elevates itself.

• Unsigned apps are recognised by their path: if one moves, or later ships signed, the choices you made for it (for example blocking it) no longer apply, and the log does not point this out. So is a signed app whose file name is not the name in its version information (a renamed copy), or that has none: it can only be allowed once. Signed Electron apps such as VS Code are the exception: their version information names Electron, and they can still be always allowed.

• The AI cannot type or paste into password fields and asks you to type instead, but a password field the app does not mark as one is not recognised.

• Typing and key presses follow the app's pace, slower than pasting; long text is sent in several calls. In some built-in UWP apps (Calculator, Settings) Deskpass cannot confirm every key was read, so the AI is asked to check with a screenshot.

• A key is occasionally lost when Deskpass falls back to bringing an app to the front; about 1 in 30 runs in testing under natural conditions.

• Background operation:

  • Not every app supports it. Web content and apps that draw their own surface (for example the Paint canvas) usually have no elements to act on, so the AI gets background_unavailable and may redo the action in the foreground.

  • A background scroll scrolls the list or pane under the point with wheel messages posted to the window there, or through UI Automation when those move nothing, not with the real mouse wheel: one step is a small increment, which need not be the distance of a wheel notch, so the AI checks the result with a screenshot. Scrolling is background by default, which is a change from earlier versions; where nothing under the point scrolls, the AI gets background_unavailable and may redo it in the foreground.

  • A background click is a UI Automation action, not a real mouse click. A few apps react only to the real mouse: the action reports success but nothing changes. The AI checks with a screenshot.

  • Ticking a checkbox in the background occasionally has no effect in a few apps; the result reports the unchanged state, so the AI can redo it.

  • A key you type in the instant between the app grabbing the foreground and Deskpass handing it back can reach that app. This is why Deskpass waits for you to stop typing first.

  • Occasionally the foreground cannot be handed back: the action is done but your window lost focus. The AI is told and does not repeat the action; click back into your window.

  • If your front window is one that cannot take focus (such as the on-screen keyboard), the window stacking order afterwards may differ from before.

  • A drop-down that the app draws as its own popup can appear on top of your window; Deskpass does not move it.

Tests

You need Node.js 22 or newer and Rust stable with the MSVC toolchain (Visual Studio C++ build tools).

npm ci
npm run build:ts
$env:DESKPASS_NO_DESKTOP = "1"   # skip the end-to-end tests that open real windows, as CI does
npm test
cargo test --manifest-path helper/Cargo.toml --bins

Without DESKPASS_NO_DESKTOP, npm test also runs the end-to-end tests, and npm run test:rust runs the desktop tests in helper/tests. Those send real input to the project's own test windows: keep the screen unlocked, leave the mouse and keyboard alone, and do not run the two suites at the same time. docs/development.md has the details.

Contributing

See CONTRIBUTING.md for building, testing and keeping both READMEs in step. Release notes are in CHANGELOG.md. To report a security problem, follow SECURITY.md.

Acknowledgements

• trycua/cua: its Windows driver's explicit background and foreground delivery modes shaped ADR 0001. Deskpass took ideas from it; no code was copied.

• Hallpass: the sibling project for Chrome, whose installer approach Deskpass follows.

License

MIT.