glovebox

Scriptable computer use for coding agents. One persistent JavaScript session, many GUI actions per call, accessibility text by default, screenshots only when the model asks.

const calc = await gb.getApp("Calculator");
await calc.pressKey("Escape");
await calc.typeText("6*7=");
await calc.getAXState();

Calculator — window 0 "Calculator" — image 460x816 — changes since last read (~ changed, + added):
~					8 scroll area id=StandardInputView;value:42
~						9 text "Edit field" value="42"

On the next call calc is still bound. That is the whole idea: an agent writes a few lines per step instead of making one tool call per click, and reads a diff of the accessibility tree instead of a screenshot.

Input goes to apps in the background, like Codex computer use: the app is not brought to the front, your real cursor does not move, and a drawn agent cursor shows where each action lands.

glovebox works with any agent that speaks MCP (Claude Code, Grok CLI, OpenCode, Codex, Gemini CLI, Cursor) and with agents that can only run shell commands, through glovebox run.

Why

Most computer-use tools give the model one tool per action and return a screenshot after each one. Every click is a model round trip, and every read fills the context with an image. OpenAI's Codex computer use showed a better shape: a persistent script session where one call does a whole step. In one real Codex session, 786 calls had a median length of about 130 characters. glovebox brings that shape to every agent, without depending on any vendor's app. See docs/design.md for the comparison and the reasoning.

Related MCP server: grok-computer-mcp

Status

What has been run for real on macOS 27 (Apple silicon), not mocked:

• glovebox smoke over MCP: one batched call types 6*7= and reads 42; the binding persists to the next call; index clicks compute 7+1=8 and the read comes back as a diff; a pixel click lands on the 9 key; the screenshot is 460x816, the same space as clicks; js_reset drops bindings.

• Claude Code (headless) attached through glovebox mcp: two calls, the second reusing the first call's binding.

• glovebox smoke --full with Firefox frontmost the whole time: background keys in Calculator; in TextEdit, a pixel click placed the caret, a drag selected a line, a scroll moved the view; the frontmost app never changed.

• The approval dialog, a blocked app (Terminal) being refused, and Escape to stop a script.

The Grok, OpenCode, and Codex configs below follow each client's documented MCP format but have not been exercised end to end yet.

Install (macOS)

Requires Node 20.10+ and the Xcode command line tools (xcode-select --install).

npm install -g github:vpatel9202/glovebox
glovebox signing --create   # optional, recommended: grants survive helper rebuilds
glovebox setup

setup builds Glovebox Helper.app into ~/.glovebox/ (about a minute), then walks you through two macOS permissions for it, one at a time, waiting while you turn on "Glovebox Helper" in System Settings > Privacy & Security > Accessibility and then Screen & System Audio Recording. It also offers Input Monitoring, which is optional and only needed for Escape-to-stop. Then:

glovebox doctor    # every line should say ok
glovebox smoke     # a real end-to-end test on Calculator

The helper is its own app so that one grant covers every agent and terminal. To keep those grants when the helper is rebuilt (for example after an upgrade), create a local signing certificate once, before setup:

glovebox signing --create

Without it the helper is signed ad hoc, and macOS treats every rebuild as a new app: clear the old grants with tccutil reset Accessibility|ScreenCapture|ListenEvent io.github.vpatel9202.glovebox.helper and run glovebox setup again.

Attach to your agent

Claude Code:

claude mcp add --scope user glovebox -- glovebox mcp

Grok CLI (~/.grok/config.toml) and Codex (~/.codex/config.toml):

[mcp_servers.glovebox]
command = "glovebox"
args = ["mcp"]

OpenCode (opencode.json):

{ "mcp": { "glovebox": { "type": "local", "command": ["glovebox", "mcp"], "enabled": true } } }

Shell-only agents:

glovebox run 'const calc = await gb.getApp("Calculator"); await calc.getAXState()'
glovebox run 'await calc.typeText("2+2="); await calc.getAXState()'

More clients, the agent skill file, and CLI details: docs/agents.md.

The API in one screen

• gb.getApp(name) launches or finds an app without stealing focus. gb.listApps().

• app.getAXState() returns the accessibility tree as text; later reads are diffs. {frames: true} adds element boxes, {disableDiffing: true} the full tree.

• app.getScreenshot() is the only way to get an image.

• app.click(target), doubleClick, rightClick, scroll, drag, setValue, focus, selectText, performSecondaryAction.

• app.pressKey("ctrl+a") uses xdotool names (super is Cmd). app.typeText("text\n").

• A target is an element index from the tree or [x, y] pixels in the window image, which is exactly the screenshot's pixel space.

Full reference: docs/api.md, or help() inside a session.

Safety

The first time a session touches an app, you get a dialog: Allow for This Session, Always Allow, or Deny. Terminals, password managers, and System Settings are blocked unless you run glovebox allow <bundle id>. To stop a running script, switch to another app; with the optional Input Monitoring permission, Escape works too. Every action is logged to ~/.glovebox/actions.log. Details and limits: SECURITY.md.

glovebox allow Calculator        # pre-approve
glovebox approval allowlist      # never prompt; only allowlisted apps
glovebox log                     # recent actions

How it works

agent ──MCP stdio──▶ glovebox mcp ──▶ kernel (persistent JS session)
agent ──shell──▶ glovebox run ──▶ daemon ──▶ kernel        │
                                                            ▼ Unix socket
                                        Glovebox Helper.app (Swift: AX, CGEvent,
                                        ScreenCaptureKit, approval dialogs)

Background input uses private macOS (SkyLight) functions, looked up at runtime with a foreground fallback; everything else is public API. No OpenAI, Anthropic, or other vendor binaries. Details: docs/design.md.

Development

npm test          # unit tests (cell rewriting, keys, diffs, coordinates)
glovebox smoke    # GUI test on Calculator
glovebox smoke --full   # adds background click/drag/scroll checks in TextEdit
GLOVEBOX_DEBUG=1 glovebox mcp   # kernel logs on stderr

License

MIT