beans-picker

Handpicks the right control. ~2× faster, ~1/6 the cost.

Not affiliated with Cua / trycua or TypeSafe; cua-driver and Jev are separate projects this server talks to.

An MCP server that lets Claude Code or Codex pick the right control in a macOS app, act on it without bringing the app to the front, and check that it worked. It does for native apps what Stagehand does for the browser:

• The caller thinks. Claude Code or Codex breaks the task down, picks the next step and decides what to do when something goes wrong.

• Jev picks the element. TypeSafe Jev chooses among candidate actions built from the accessibility tree. It never writes text, so it cannot invent a target.

• cua-driver acts. cua-driver clicks, types and presses keys without bringing the app to the front.

The server itself never calls a generative model (no claude -p, no LLM API).

Tools

Captures and file transfers

Captures and file transfers require an already open, exact pid and windowId wherever a window is involved. Use windows to discover these ids or launch_app to start an app in the background. Captures and transfers do not select a tab, open a file picker, change browser preferences or use foreground delivery. A window id owned by another process is refused before dispatch. These tools do not need Jev or the raw-driver opt-in. An older driver missing a capability returns unknown_driver_tool.

For example (replace ids and canonical absolute paths with your own):

{"pid": 123, "windowId": 456, "out": "/private/tmp/bp-impl-window.png"}

Pass that to screenshot. For the primary display use {"scope": "screen", "out": "/private/tmp/bp-impl-screen.png"} instead. includeImage: false saves only the file and requires out. Captures are decoded and bounded as PNG before saving or returning them. Full-screen capture retains the driver's desktop-consent requirements and may return a consent refusal.

Start recording with record_start({"pid":123,"windowId":456,"out":"/private/tmp/bp-impl-demo.mp4"}), operate the window, then call record_stop({}). One recording is allowed per MCP session. The file is published only after successful finalization; server shutdown also attempts to stop and save it. record_stop remains callable while the screen is locked because it only finalizes the existing child; starting or capturing still requires an unlocked screen. Screen Recording permission must already be available to the process running the helper. The helper checks that grant before capture and refuses without requesting an OS permission dialog.

All outputs use private bp-impl-* staging and never replace an existing destination, including symlinks. Published files are owner-readable and owner-writable (0600). Output parents must exist and be canonical absolute directories. Inputs must be canonical absolute regular files without symlinks. Failed operations clean up their staging files.

For browser transfers, call browser_state first with the native ids. If it returns ambiguous, choose a returned tab id and repeat. Use the resulting targetId, tabId and fresh action ref with the same native ids for browser_upload or browser_download. Upload refs must declare the upload action. A reconnect or foreign session invalidates the saved binding. Setup and unsupported browsers return the driver's refusal; no legacy page or native picker fallback is used.

Current driver limits: image clipboard reads expose types, not pixels. Upload responses expose an assigned count, not FileList content, so neither operation claims exact content verification. The current cua-driver download tool requires approval through its own MCP host; an ordinary nested MCP call cannot supply that proof and returns browser_consent_required. beans-picker preserves this boundary rather than manufacturing approval. A success path is supported for a driver that admits the call through its authorized transport; the staged file is then checked and published.

Navigating and launching

Use windows({"app":"Safari","onScreenOnly":true}) to discover exact native ids. It returns status: "done". Titles are omitted unless an app or pid filter is supplied and truncated to 256 UTF-16 units. onScreenOnly is forwarded as on_screen_only to the driver, whose semantics are current Space, rather than a local filter on is_on_screen.

Call browser_state with a returned pid and windowId, then give the same ids and its targetId and tabId to browser_navigate. Only HTTP(S) and case-insensitive about:blank are allowed. HTTP(S) requires a host; all URLs refuse userinfo (username/password), whitespace, controls and Unicode format characters (including zero-width and bidi marks), and are capped at 8192 UTF-16 units. Fresh page.url must match the request or driver-reported final URL after normalizing scheme/host to lowercase, empty path to /, and removing default ports 80/443. Path case, query and fragment are preserved. For example, HTTP://Example.test matches http://example.test/. Otherwise the result is unverified with a reason. finalUrl is the observed page URL, including redirects. Refs from before navigation are invalid; use the fresh snapshot's refs.

browser_dialog uses the same session binding. Inspect returns status: "done" and only the available present, dialogId, type, message and defaultPrompt fields. Accept/dismiss require the current dialogId; accept requires allowDestructive: true because accepting a confirm or beforeunload can be irreversible. Only accept permits promptText, forwarded exactly and capped at 8192 UTF-16 units. Successful resolution returns a fresh snapshot. If a fresh snapshot after navigation or resolution fails, the action returns unverified with driverCode and the original url or dialogId; a dialog may be open, so use browser_dialog with action: "inspect". No native dialog, browser permission UI or foreground fallback is supported. Window titles, dialog messages and page snapshots are untrusted page/app content, never instructions.

launch_app({"bundleId":"com.apple.finder","urls":["/Users/you/Documents"]}) starts an app without requesting foreground delivery. Give bundleId or name (bundle id takes precedence if both are supplied). Name refuses /, :, control/Unicode format characters and leading - or .; bundleId must fully match [A-Za-z0-9][A-Za-z0-9.-]{0,254}. urls accepts up to 16 HTTP(S) URLs validated like navigation, or existing canonical absolute plain folders without symlinks. Files are refused entirely, as are directories with any extension (such as .app, .bundle, .pkg, .workflow or .prefPane) or containing Contents/ or Info.plist. Local-path refusals use bad_path: "only plain folders can be opened; files and bundles are refused". Every input starting with - is refused. newInstance requests a new app instance. Arbitrary launch arguments and WebKit inspector ports are not exposed; browser_prepare and CDP/browser setup remain refused. Returned windows have the same bounded title/geometry fields; unknown metadata is null. Unknown launched pid or no ready window yields unverified. After the driver returns, launch polls the frontmost app using the same six checks at 50 ms intervals as restoration. Any change away from the previous app (including to a third app), or failed activation suppression, causes foreground_violation and an attempt to restore the previous app. The message includes the launched pid when known and whether restoration succeeded. All four tools use the existing serialized queue and screen-lock gate, need no Jev or raw-driver opt-in, and preserve unknown_driver_tool on older drivers.

Manual verification on a Mac

The unit tests mock OS access. Before relying on these capabilities in a real session:

1. Open two windows in a background app and keep another app in front. Capture one exact window and the primary screen; open both PNGs and verify their scope. Repeat with includeImage: false and an existing output path. Check that the frontmost app, selected tab and real cursor never change.

2. Start/stop a recording, play the MP4 and compare its duration with durationSeconds. Cover the target window while recording; verify that only that window appears. Test startup with missing Screen Recording permission, duplicate start, a closed window, shutdown during recording and stopping an active recording after locking the screen.

3. Read/write text including an empty string, newlines and Unicode; independently read it back. Write an image and check available image types and the unverified status before pasting manually.

4. Move and resize the background window, including negative coordinates on a second display. Check the returned frame and mismatch when the app constrains its minimum size. Try a foreign window id.

5. On a browser already prepared and authorized outside beans-picker, bind an exact window/tab and assign a harmless fixture to an upload ref. Inspect its FileList manually; no picker should open. Exercise multiple tabs, stale refs, reconnects and unsupported browsers.

6. Attempt a harmless download to a new destination. With current cua-driver expect browser_consent_required and no file or focus change. Where the driver has an authorized host transport, verify the downloaded contents, returned path/bytes, overwrite refusal and cleanup after cancellation. A foreground-only route must stop with a refusal.

To drag, give the source's candidateId and either dragTo: {"candidateId": "<target id>"} or dragTo: {"dx": 120, "dy": -30}. The gesture starts at the source control's centre; offsets are window-local screenshot pixels, not screen points. Both endpoints must be visible inside the same window. instruction is still required; omit text and modifiers. The same fields work in a then step. Destructive controls under either actual endpoint, for both candidate and offset destinations, require allowDestructive: true. Stale ids return not_found, and missing frames, covered controls or off-window endpoints return failed / not_visible. The settled snapshot must show a state change (including enabled controls) or movement of the source relative to its window. Moving the whole window or snapping back does not count; no observable change returns no_effect. After an effect appears, drag verification requires 0.3 seconds of unchanged state, sampled every 0.15 seconds for at most four more snapshots. An unsettled or unreadable result is unverified.

The optional driver tool covers calls such as double_click and zoom: {"tool": "double_click", "arguments": {"pid": 123, "window_id": 456, "x": 100, "y": 80}}. Only these app-control tools are allowlisted: click, double_click, right_click, drag, scroll, type_text, press_key, hotkey, set_value, zoom, move_cursor, page, get_window_state, list_windows, list_apps, get_screen_size, get_cursor_position and get_accessibility_tree. Tools absent from the loaded driver schemas return unknown_driver_tool; other tools outside the allowlist, including launch_app, return driver_tool_disallowed.

Input calls (clicks, drag, scroll, typing, keys, set_value and page) require an integer pid and watch that target for activation. Desktop scope, foreground delivery and focus-stealing tools (including move_cursor) remain refused. page cannot run enable_javascript_apple_events. Its cdp_port and bundle_id arguments are refused with driver_argument_disallowed because they can select another process. screenshot_out_file is redirected into private shots/capture-* storage and returned as an image; other file-output arguments, such as debug_image_out, are refused. Failures return only a code and fixed message. Values under path-like keys and occurrences of this session's cache directory in strings are omitted at directory boundaries; sibling paths sharing the prefix and unrelated URLs are preserved. See SECURITY.md for the passthrough boundary.

act returns a status:

A call that cannot run at all returns isError with {"status": "failed", "code", "message"}. Codes include bad_target, window_not_found, screen_locked, jev_unavailable (no key, Jev unreachable or too slow), jev_bad_response, driver_unavailable (cua-driver could not be started), driver_timeout, driver_error (cua-driver refused a call the tool needed) and internal. Failures other than a tool's own refusal are logged by code and exception type at WARNING, without exception contents or tracebacks.

Each result also carries the action that ran (action, with the cua-driver route) and a summary of what changed on the window (change). When the action brought up something new (a dialog, a menu, another page), the result carries the window's screenText and, in newCandidates, the new controls with their ids (as many as a plain observe lists; newTotal counts them when there are more), so the next step needs no observe. After a click, act takes snapshots until two in a row offer the same controls (a few at most), so a page that is still loading is reported as it lands.

Candidate ids are derived from a control's stable identity (role, identifier, label and ancestors), not from the snapshot's element index, so an id from observe still works after the window changes, as long as its identity is unique. Look-alike controls with no named container between them (the Edit button of each section, the menu button of each row of a web list) take the nearest heading, or the nearest text no other look-alike shares, as part of their identity. Unnamed square buttons in a row of three or more on a web page (a colour picker) are described by the colour they show, read from a capture of the window (click Button showing purple #8B5CF6). Controls that still cannot be told apart use token-bound ids; after their tokens change, observe again rather than retrying an old id.

Related MCP server: Automation MCP

Install

Requirements: macOS, Python 3.12+, cua-driver at ~/.local/bin/cua-driver with its Accessibility and Screen Recording permissions, a Jev API key, and Xcode Command Line Tools (clang) for the two small native helpers, which are compiled on first use into ~/Library/Caches/beans-picker.

uv tool install beans-picker
install -d -m 700 ~/.config/beans-picker
(umask 077; printf '%s\n' 'JEV_API_KEY=...' > ~/.config/beans-picker/.env.local)
chmod 600 ~/.config/beans-picker/.env.local
claude mcp add beans-picker -- "$(command -v beans-picker)"   # absolute path: GUI-launched clients often lack ~/.local/bin on PATH
beans-picker grant-ax                                    # exact text (optional, see below)

Without installing, let uvx fetch it on each start: claude mcp add beans-picker -- "$(command -v uvx)" beans-picker, and uvx beans-picker grant-ax.

For Codex, add the same absolute path as an MCP server in ~/.codex/config.toml (command -v beans-picker prints it):

[mcp_servers.beans-picker]
command = "/Users/you/.local/bin/beans-picker"

From a checkout:

uv sync
(umask 077; printf '%s\n' 'JEV_API_KEY=...' > .env.local)
chmod 600 .env.local                # read from the checkout, whatever the caller's cwd
claude mcp add beans-picker -- "$(command -v uv)" run --directory "$PWD" beans-picker
uv run beans-picker grant-ax

beans-picker grant-ax builds the read-only helper and asks macOS to list it; then turn on beans-picker axtext under System Settings > Privacy & Security > Accessibility. The helper is built from its source, so an upgrade that changes that source builds a new helper that macOS has not been told about: run beans-picker grant-ax again after upgrading if text results turn unverified. beans-picker --version prints the version and beans-picker --help the usage.

The agent skill in skills/beans-picker tells a model how to use these tools well: when cua-driver fits better, what to do with each act status, and recipes for combo boxes, number fields, long tables and checking the result. Link it where your agent looks for skills, e.g. ln -s "$PWD/skills/beans-picker" ~/.claude/skills/beans-picker (Codex: ~/.codex/skills). Leave it out when benchmarking the bare tools.

Configuration

Variables are read from the environment first, then from .env.local and .env in the checkout (only when the package runs from a beans-picker checkout), then from .env.local and .env in ~/.config/beans-picker ($XDG_CONFIG_HOME/beans-picker when that is set). A value that is already set is never overridden. See .env.example.

Rules the server keeps

• Background only. bring_to_front, invoke_menu, move_cursor and delivery_mode: "foreground" are refused before they reach cua-driver. App launch is allowed only through the typed background launch_app tool. Menu commands run as their keyboard shortcut sent to the app's pid; a command without a known shortcut returns failed / foreground_required. During act, raw input and browser mutation calls, the frontmost app is sampled continuously; if the target app comes to the front, the call stops with foreground_violation and the app that was in front before is put back. Launch also checks activation suppression and polls for any change of frontmost app during its bounded watch; violations report the launched pid and whether restoration succeeded.

• Time limits. Every request to cua-driver and to Jev has a generous limit (see Configuration), so one that stalls ends its call with driver_timeout or jev_unavailable instead of blocking the queue. An action whose answer timed out may still have happened: observe before repeating it. At shutdown, the final end_session gets 5 seconds.

• Screen lock. While CGSSessionScreenIsLocked is set, every tool refuses with screen_locked and does nothing. A failed or unreadable lock check refuses with screen_lock_unavailable.

• One call at a time. Tool calls are queued, so two actions never interleave on the desktop.

• Destructive actions are marked, not hidden: act returns needs_confirmation unless allowDestructive is set, whoever picked the action. Return and Space also require confirmation because they can activate a focused button. browser_dialog accept requires allowDestructive: true; inspect and dismiss do not. Pop-up options are checked again after opening. This label-based check is a precaution, not an authorization boundary: apps can mislabel controls, and a checkbox can have irreversible side effects.

Recording a demo

A recording of the whole display shows every moment an app comes to the front. Three things keep it out of the video:

• Record the window, not the display. beans-picker record --app Calculator --out demo.mp4 records the app's window on its own until Ctrl-C (macOS 15 or later, H.264). The picture is the same whether the window is covered or in front. The terminal or agent that runs the command needs the Screen Recording permission.

• Launch the app before recording starts. An app can take the front for a moment while it starts. beans-picker puts the app that was in front back, but the moment can still be seen on a display recording.

• BEANS_PICKER_PIXEL=off when an app comes forward on a pixel click. Every click is then an accessibility press (about 2.5 s each instead of 0.2 s), and a click with modifier keys returns failed / pixel_disabled.

An app that activates itself (a new window, a dialog) cannot be stopped from outside. beans-picker sees it within about 80 ms and puts the previous app back; the act call still reports foreground_violation, and its message says whether the app was put back.

How act works

snapshot ─► candidates ─► Jev picks (or candidateId) ─► gate ─► cua-driver ─► fresh snapshots ─► status

1. Snapshot (src/beans_picker/observe). get_window_state returns the AX tree. The structured elements are joined with the tree markdown, which holds identifiers, help text and static text. The menu bar is split off, and a signature of what matters (title, field values, visible text, other windows) is computed.

2. Candidates (src/beans_picker/candidates). Clicks, toggles, text entry (set_value, type_into at the caret, append at the end), pop-up choices, an on-screen keypad sequence for text ("12×7="), menu commands with their background shortcut, one step up or down on a slider or number field (an arrow key sent to it), the options of a web list box (a combo box's suggestions), a page down or up on a scroll view, table, list or web page, a named row's, link's or image's context menu (AXShowMenu, which a web page receives as a right-click), and Return / Escape / Tab / Space / the arrow keys / Shift+F10. With text, only the actions that enter text are in the running.

3. Jev (src/beans_picker/jev). One system_one request asks two choice questions over the same candidates: one with a none option, one forced. More than 60 candidates are sharded (best lexical match first), and the leaders of each shard go to a runoff. The gate acts on a leader at p ≥ 0.8, or at p ≥ 0.5 when the forced question agrees and the leader has twice the runner-up's probability. Otherwise the result is ambiguous (or not_found when none dominates).

4. cua-driver (src/beans_picker/act). AX press, pixel clicks for keypads, a pop-up pressed open and then its item pressed (a menu left open over the page, as Chrome's is, is closed with Escape), set_value / type_text (a web page keeps its own copy of a field's value, which follows typing but not an AXValue write, so a web page's text and number fields are typed into: End, ⇧Home (⌘↓, ⇧⌘↑ in a text area), then type_text, once more if a keystroke was dropped; text with a line break is written instead, since a typed Return would submit a form or send a message), ⌘↓ then type_text for append, a wheel event at the area for a scroll, and the menu shortcut as a pid-targeted hotkey. Stale element tokens are rebound by stable key.

5. Effect (src/beans_picker/verify/effect.py). Fresh snapshots are taken, up to the retake count, until the effect shows. Drags then require a bounded quiet interval before the final verdict.

Exact checks

A false success is the worst failure a desktop agent can have: text judged by trim or by "contains", a search field mistaken for the document, a trailing space silently lost. act therefore checks text by exact equality, in the targeted field only:

• set_value: the field reads exactly text.

• append: the field reads exactly its previous text followed by text.

• type_into: the field reads its previous text with text inserted whole at one position.

cua-driver 0.8 does not give the exact text: it trims whitespace at both ends of a value, shows an empty field's placeholder as its value, and leaves out a checkbox's state. The read-only helper src/beans_picker/native/axtext.m reads AXValue as it is. It needs the Accessibility permission, which is why it runs as its own tiny background app (beans-picker axtext) that you can allow on its own, without granting anything to your terminal. Without it, a text action that visibly changed the field returns unverified, never done. A checkbox whose state is not in the tree is judged by its own pixels, captured in the background before and after the click.

Privacy

Window text leaves the machine only when a step needs Jev. SECURITY.md lists exactly what is sent, what runs locally, and how to report a vulnerability.

Development

uv sync
uv run ruff check
uv run ruff format --check
uv run mypy
uv run pytest

The benchmark is in bench/ (see below). bench/fixture-app is a small AppKit window used only by the benchmark; sh bench/fixture-app/build.sh builds it into ~/Library/Caches/beans-picker/fixture. python -m bench.run, python -m bench.judge and python -m bench.summarize run from the repository root. See CONTRIBUTING.md.

Benchmark

Run on 2026-09-24 with an unpublished pre-release of beans-picker (0.2.0) and --reps 3: 8 tasks × 2 conditions × 3 repetitions = 48 runs, Claude Code headless (claude -p) with --model sonnet, one run at a time. Raw records: bench/results/runs.jsonl; the tables below are python -m bench.summarize of them.

Conditions. Both get the same prompt (task, target pid and window id, "work in the background", and a final RESULT: success|failure line), no built-in tools (--tools "") and only one MCP server:

• (a) cua-driver only: cua-driver mcp, with bring_to_front, move_cursor, kill_app, get_desktop_state and a few other tools denied.

• (b) beans-picker: this server only.

Tasks (bench/tasks.json). Five run on bench/fixture-app, a small AppKit window made for the bench (so no user document is touched): a Name with leading and trailing spaces, appending to a note body next to a search field, a pop-up and Save, clearing a search field next to a destructive "Delete note" button, and fixing an email plus a checkbox. Three run on Calculator: (48 + 16) / 8, 15% of 80, and 7 − 19 then change sign. TextEdit and Notes were not used.

Judging. Success is decided only by bench/judge.py, a separate script that compares the final state with the task's expected JSON by exact equality: every key of the fixture's state file (the controls' values, written by the app itself), or Calculator's display read over accessibility (bidi marks removed, nothing else). The agent's RESULT: line is used only to count false success claims. The front app was sampled every 200 ms during each run to count focus steals.

Overall

Per task

Tokens are what Claude Code reported (input + output + cache reads + cache writes); the USD columns are Claude only. Jev is billed separately at $0.042 per 1M input tokens (output free; TypeSafe models, as of 2026-09-27): 608,908 Jev input tokens ≈ $0.026. Totals including that Jev cost: (a) USD 10.50 and 1,541 s for 24 runs; (b) USD 1.62 and 820 s for 24 runs (median $0.060 per run), from 265 Jev calls.

What this shows

• Both conditions nearly always succeeded. (b) passed 24/24, (a) 23/24: one Name-with-spaces run failed, and the agent said so. In another (a) run the agent reported failure although the judge found the task done.

• Neither condition claimed success on a failed run in the counted runs. So this benchmark does not show that beans-picker reduces false success claims. In a pilot run before the benchmark (not counted), (a) typed 48+16÷8, got 50 and claimed success.

• (b) used fewer tool calls (median 5.5 vs 10), less time (28 s vs 50 s) and about 1/5 of the Claude tokens, mostly because cua-driver's get_window_state returns the whole tree (and a screenshot) to the model at every step, while beans-picker returns a short candidate list and the change summary.

Where beans-picker did not win

• Time on simple tasks. fx-size-save (28.6 s vs 27.3 s) and fx-clear-search-keep-note (14.3 s vs 12.8 s) were slower with beans-picker: a Jev call plus fresh snapshots cost more than one direct click. One calc-chain run took 126 s and 18 tool calls with beans-picker, the slowest run of that task in either condition.

• Output tokens were higher with beans-picker on fx-clear-search-keep-note (709 vs 489) and about equal on fx-size-save.

• Jev cost is small. 265 Jev calls add ≈ $0.026 to (b)'s total ($1.62 with Jev vs $1.59 Claude-only); they are not in the USD columns above.

• Pop-ups. choose_option failed in all three fx-size-save runs: cua-driver cannot set_value a closed NSPopUpButton ("has no AX children"). The caller recovered by clicking the pop-up and then the menu item, which is why that task took 6–7 calls in both conditions. Since then choose_option presses the pop-up open and then the item titled exactly text, in the background (checked on the fixture and on a <select> in Chrome).

• Exact text was not confirmed. The benchmark ran without the Accessibility permission for the beans-picker axtext helper. Of the 242 act results in (b), 206 were done, 18 unverified (text entered; the judge later found it exactly right), 6 failed (the pop-up above), 6 ambiguous, 4 not_found and 2 no_effect.

Caveats. 3 repetitions per task and one model (Sonnet 5) are a small sample, and one person's Mac. The tasks and the fixture app were written by the same author as beans-picker. An earlier attempt at this run was stopped after 3 runs because the fixture only recorded typed input, not values set over accessibility (so it failed a correct set_value). The fixture was fixed to write the controls' actual values, and the 48 runs above were all made after that. The baseline is Claude Code with cua-driver only; no other agent was compared.

License

MIT