planchette

No API key: the agent (Claude Code, Codex, Cursor, …) is the loop, through a small CLI of window-control primitives.

Platforms

• macOS — the polished path: per-window capture, live-overlay picker.

• Windows / Linux (X11) — portable backend (mss + pynput + pywinctl); select windows with pick --name / pick --list instead of the overlay.

• Wayland — unsupported (it blocks synthetic input and capture by design).

Requires Python ≥3.10.

Related MCP server: WinPilot Computer Use MCP

Install

# 1. Put the `planchette` CLI on PATH
curl -LsSf https://raw.githubusercontent.com/shahriarshm/planchette/main/install.sh | sh
#    (or, if you already have uv: `uv tool install planchette`)

Then wire it to your agent:

# 2a. Claude Code: install as a plugin
#     /plugin marketplace add shahriarshm/planchette
#     /plugin install planchette@planchette-marketplace
# 2b. Any MCP-capable agent (Codex, Cursor, …): register `planchette mcp`
#     — see "MCP mode" below.
# 2c. Any other agent tool: point it at AGENTS.md in this repo —
#     it documents the capture → look → act loop.

MCP mode — any MCP-capable agent

planchette mcp serves the same primitives over MCP (stdio). Screenshots come back inline in the tool result, so the agent needs no file access — this is the recommended door for Codex (no sandbox bypass needed), Cursor, and anything else that speaks MCP.

Then ask the agent to pick a window (pick_window, or pick_window name="Telegram") and drive it. macOS permissions (Screen Recording + Accessibility) must be granted to whatever app hosts the agent.

Skill mode — agents that read SKILL.md / AGENTS.md

The skills/control-window/ folder is a standard Agent Skill: the same folder works in Claude Code, Codex, Cursor, and other SKILL.md readers.

Required macOS permissions

Grant these to the app you run your agent from (Terminal, iTerm, VS Code…), in System Settings → Privacy & Security:

1. Screen Recording — to capture the target window (else screenshots are black).

2. Accessibility — to move the mouse, click, type, and raise windows.

After granting, fully quit and reopen the terminal app.

Use

In Claude Code:

/planchette open a new tab and search for the weather

…or just ask in chat: "control my Telegram window and message Saved Messages 'hi'." Claude runs the picker (hover + click the window you want), then loops: screenshot → decide → act, until the goal is done.

• Background control (macOS): input is delivered straight to the target app's process — the window needn't be frontmost, your focus and cursor stay put, and you can keep chatting with Claude in the terminal while it drives. (Exceptions: modifier combos briefly activate the target and hand focus back; scroll hops the cursor there and back.) On Windows/Linux input follows real focus — don't fight it while it runs.

• One display in v1.

CLI (what the agent drives)

planchette pick                    # live-overlay picker (macOS) → select a window
planchette pick --list             # print candidate windows
planchette pick --name "Telegram"  # select frontmost match by app/title substring
planchette capture [--out PATH]    # screenshot it → PNG, prints "PATH  WIDTHxHEIGHT"
planchette capture --crop X Y W H  # native-res zoom of a region (for small text)
planchette click X Y [--double] [--right]  # X Y are captured-image pixels
planchette drag X1 Y1 X2 Y2        # select text, sliders, drag & drop
planchette move X Y
planchette type "text"
planchette key "cmd+t" [--repeat N]
planchette scroll X Y up|down [N]
planchette raise
planchette status
planchette agent [--agent claude|gemini]  # macOS: pick a window → floating panel drives it
planchette mcp                     # serve window control over MCP (stdio)
planchette hotkey [--combo C]      # macOS: global hotkey (default ctrl+cmd+p) → agent panel
planchette hotkey --install        # …as a login LaunchAgent; --uninstall removes it

Window bounds are re-read from the live window before every action, so a window that moved after pick still gets clicked in the right place.

Global hotkey → agent panel (macOS)

planchette hotkey --install registers ctrl+cmd+P system-wide (like the ⌘⇧5 screenshot overlay) and keeps it across logins. Pressing it runs planchette agent: pick a window under the overlay, then a small floating panel appears — type a goal ("reply ok to the last message") and a headless agent session (claude or gemini, whichever is installed — force one with planchette agent --agent gemini) drives that window, streaming its actions into the panel. Enter again for follow-ups in the same session; Esc closes it. A dropdown on the input row lists the installed agent CLIs — switch anytime between turns; switching starts a fresh conversation for the new agent.

The spawned session is scoped to window control only (claude: --allowedTools "Bash(planchette:*)" Read; gemini: a generated ~/.planchette/.gemini/settings.json capping its tools the same way). The agent CLI and planchette must be installed; the panel finds them on PATH plus the usual install dirs (~/.local/bin, /opt/homebrew/bin, /usr/local/bin). Gemini also needs auth the spawned process can see, e.g. GEMINI_API_KEY in ~/.gemini/.env.

Note: the LaunchAgent runs outside your terminal, so macOS asks for Accessibility again — grant it to the listed Python in System Settings → Privacy & Security; the daemon retries on its own. Pre-picking for a terminal session still works: pick via the hotkey, then just Esc the panel.