Assistant Vision

A light Windows desktop application that gives an assistant fresh visual feedback from one selected window, including the results of its mouse and keyboard actions.

Preview, not a production automation guarantee. The working implementation combines Tauri/Rust capture with either MCP or a separate Codex App Server session. It does not attach a live feed to an existing Codex desktop conversation.

Try it

Prerequisites: Node.js 22+, stable Rust, Tauri platform prerequisites, and an installed, signed-in Codex CLI for App Server mode.

npm ci
npm test
cargo test --manifest-path src-tauri/Cargo.toml
npm run build
npm run app-server

After building on Windows, double-click Start Assistant Vision.cmd instead of typing the last command. This is a repository launcher, not an installer; it needs the repository's dependencies and compiled executable.

1. Open the test window, refresh the list, select Vision Test Surface, and share it.

2. Enable Allow mouse & keyboard if the task requires input.

3. Enter a task in Work with Codex and click Start task. On Windows, this attempts to bring the selected target forward. Keep that window in the foreground.

4. Read the response and progress checkpoint in the controller. The preview updates automatically.

5. Use Stop sharing & control to revoke capture/input. Closing the controller also closes its connected session.

No refresh button is needed after clicks or typing. The refresh-list button only refreshes available window titles.

How the assistant sees

Every task starts with a fresh image. During an active task, changed images are delivered automatically at most once per second, keeping the newest change instead of building a queue. Each input tool also returns a fresh image for verification.

Codex is instructed to report the sequence it inspected using vision_checkpoint, along with completed work, uncertainties and the next step. Automatic delivery waits for that report before sending more. This acknowledgement is a model report, not proof of comprehension or successful business processing.

Near the visual budget, further observation/input is refused until the assistant checkpoints and finishes its turn. If it requests continuation, the controller asks App Server to compact the conversation, waits for actual completion, then starts again with the original task, checkpoint and a fresh image. A stopped or changed target prevents automatic continuation. Reduce context also requests compaction while idle. A checkpoint survives compaction but is labelled as previous until reverified.

Bounded storage

The app does not save screenshots to disk. Unchanged pixels are deduplicated before encoding; current and delivered raw pixels share buffers where possible. Stop clears local caches as pending operations finish.

These are not total RAM limits: raw pixels, temporary encoding/JSON buffers, WebView, Node and Codex use additional memory. Compaction reduces model context; it does not delete images already received by a client/provider. The app does not control provider retention or diagnostics.

Four screenshots are only the recent visual buffer. The textual checkpoint and original task carry progress across compaction; they can contain model mistakes and must be checked against the current screen.

MCP connection

Configure an MCP client to run Node with the absolute path to connector/server.mjs. It launches its own controller and permissions. MCP returns images on tool calls; automatic App Server steering and vision_checkpoint are specific to App Server mode.

Only the controller can grant sharing/control. Actions require a fresh, unconsumed frame token, identical pixels/geometry and the selected foreground window. The preview does not replace action tokens. A failed post-action observation means the result is uncertain; never repeat the input blindly.

Escape or three middle-button presses within one second request emergency stop on Windows. Already sent input cannot be undone. Stop cannot recall images already transmitted.

Scope and limitations

• App Server owns a separate ephemeral thread, with a read-only Codex sandbox. Unsupported approval requests are refused. This does not remove all Codex built-in capabilities; selected-window controls apply to Assistant Vision tools.

• Sampling targets 200 ms locally; model delivery and reasoning are slower. Brief transitions may be missed. Blinking cursors/animations count as changes and may cause conservative action refusals.

• Keyboard supports text, Tab, Enter, Backspace, Delete, arrows, Home, End, PageUp and PageDown. No modifier shortcuts, scrolling, dragging or input batches.

• A post-action image is captured after 120 ms; it does not prove the application has settled. OS focus checks and input are not atomic.

• Long-running real workloads, CPU/battery use, DPI changes, overlays and elevated/protected apps need broader validation.

• macOS compilation is checked in CI, but native capture/permissions are unverified and input is disabled outside Windows.

• No signed installer, auto-update, OCR or accessibility-tree extraction is supplied.

A real Windows demonstration completed two clicks, typed VISION TEST, pressed Tab and verified the result using 11 transmitted images (about 0.74 MiB). This is a bounded smoke test, not evidence of reliability for arbitrary applications.

See architecture and verification. MIT licensed. Use synthetic data in contributions; never commit private captures or credentials.