gpui-mcp

Give MCP agents eyes and hands inside a GPUI app.

Agents read the live UI as a semantic tree, then click, type, hover, drag, scroll and wait on real state. They can also take screenshots and diffs, record video, measure frame cost, and edit HTML-authored interfaces while the app runs. It works on Windows 11, macOS, and Linux.

Setup

Install the MCP server and add it to your MCP client:

cargo install --git https://github.com/themixednuts/gpui-mcp --locked gpui-mcp-server

{ "mcpServers": { "gpui": { "command": "gpui-mcp" } } }

Add the bridge and its GPUI build to your app:

[dependencies]
gpui = "=0.2.2"
gpui_platform = { git = "https://github.com/zed-industries/zed", rev = "16c9aa7ea6d897a8044d9501cde1b295256722f2", features = ["font-kit", "wayland", "x11"] }
gpui-mcp = { git = "https://github.com/themixednuts/gpui-mcp", branch = "main" }

[patch.crates-io]
gpui = { git = "https://github.com/themixednuts/gpui-mcp", branch = "main" }

[patch."https://github.com/zed-industries/zed"]
gpui = { git = "https://github.com/themixednuts/gpui-mcp", branch = "main" }

Both patches are required. They keep your app, gpui_platform, and the bridge on one GPUI build. They can go away once the additions in the vendor patch inventory land upstream.

Install the bridge when you open a window:

use gpui_mcp::{AppId, BridgeConfig, BridgeHandle};

let bridge = BridgeHandle::install(
    window,
    cx,
    BridgeConfig::new(AppId::new("my-app")?, "My App"),
)?;

That is the whole integration. The client discovers running apps on its own. See the demo for a complete window.

Related MCP server: Electron MCP Server

Making your UI agent-ready

The tree is built from your rendered elements and their accessibility data. Most problems come from the items below.

1. Keep the BridgeHandle alive. Store it in your root view. Dropping it stops the bridge.

2. Give elements an .id(...). Only elements with an id become nodes. Text inside an element without one becomes part of its nearest ancestor's label, and that text cannot be clicked, waited on or asserted on separately. Clickable elements already need an id in GPUI.

3. Keep ids unique among siblings. When ids collide, the nodes get longer identities qualified by their path, and exact duplicates are dropped with a DuplicateId diagnostic. Find children through parent; don't rely on the shape of an id.

4. Name controls that have no text. An icon button's label is otherwise its glyph, such as ⚙. Use .aria_label("Settings"), and use .role(...) when the role can't be inferred, as with tabs.

5. State disabled and read-only explicitly. enabled comes only from .aria_disabled(true). A grey control with no click handler still reports enabled: true. Mark read-only inputs with .aria_read_only(true).

6. Redact secrets. .frame_redacted(true) on an element with an id withholds its text and value from the bridge, and from any labels derived from them.

7. Check the diagnostics. get_ui_tree returns a diagnostics list that reports omitted, duplicate and orphaned nodes.

div()
    .id("settings")
    .aria_label("Settings")
    .on_click(cx.listener(|this, _, _, cx| this.open_settings(cx)))
    .child(svg().path("icons/settings.svg").size_4())

For HTML-authored interfaces that agents can also edit live, see the visual builder guide and the showcase.

What agents can do

Prefer the element tools over coordinates. All coordinates are logical pixels relative to the window.

GPUI Kit, gpui-pre and gpui-ce

Besides Zed's own GPUI, the bridge works with two GPUI releases on crates.io:

Each needs a small set of GPUI patches, which this repository provides. The recipes below pull in a patched copy from here.

GPUI Kit 0.7.0 (gpui-pre 0.3.7)

Add this to your workspace's Cargo.toml:

[dependencies]
gpui-kit = "=0.7.0"
gpui-mcp = { git = "https://github.com/themixednuts/gpui-mcp", branch = "main", default-features = false, features = ["gpui-pre"] }

[patch.crates-io]
gpui-pre = { git = "https://github.com/themixednuts/gpui-mcp", branch = "main" }

Call gpui_kit::init(cx), open the window with gpui_kit::open_window, and install the bridge as usual. The bridge accepts Kit's Window and App directly. To see it working, run the Kit demo:

cargo run --manifest-path examples/gpui-kit/Cargo.toml

gpui-ce 0.2.2

Add this to your workspace's Cargo.toml:

[dependencies]
gpui-ce = "=0.2.2"
gpui_ce_platform = "=0.1.0"
gpui-mcp = { git = "https://github.com/themixednuts/gpui-mcp", branch = "main", default-features = false, features = ["gpui-ce"] }

[patch.crates-io]
gpui-ce = { git = "https://github.com/themixednuts/gpui-mcp", branch = "main" }

gpui-ce is imported as gpui, so install the bridge as usual when you open a window.

Another version, or your own GPUI patches

The [patch] recipes above always give you the version this repository vendors. If your app is on an older supported version, or you carry GPUI patches of your own, keep your own patched copy instead:

1. From a checkout of this repository, write a patched copy into your app. Pass --crate gpui-ce for gpui-ce.

   cargo xtask vendor --crate gpui-pre --version 0.3.5 --output /path/to/your-app/vendor/gpui-pre

2. Apply your own patches to that copy, if you have any.

3. Point your workspace at it:

   [patch.crates-io]
   gpui-pre = { path = "vendor/gpui-pre" }

The patches are in vendor/patches/, one folder per crate and version. Each version has two:

• automation.patch has everything the bridge needs.

• font-fallback.patch is an unrelated font fix. Add --without font-fallback to skip it.

CI builds and tests the bridge against every supported version, so these patches are kept working.

Things to know

• Use one backend per app. Every gpui-mcp dependency in a GPUI Kit or gpui-ce app needs default-features = false. Zed apps that turn off default features must add features = ["zed"].

• Kit components that publish accessibility info work with all the tools. Custom-drawn components only show what they annotate.

• In Kit 0.7.0, disabled controls report enabled: true and read-only inputs don't report read_only. Kit doesn't publish these states yet.

• gpui-mcp-html only supports the Zed backend for now.

Measuring frame cost

Injected input costs the same as real input. The server waits for the frames that input caused and adds none of its own.

Call mark_frames, perform the interaction, then call get_frame_report. For every frame since the mark it gives:

• the full Window::draw time (draw_ms), split into the app's share (app_draw_ms) and the bridge's (bridge_ms);

• p50, p95 and max for each;

• every view that rendered and why, plus the cached views that replayed instead.

The render causes are:

• notified: the view, or a view inside it, called cx.notify()

• ancestor_rendered: a cached view around it rendered

• refresh: the window was refreshed

• first_draw: the view had nothing cached yet

• layout_changed: its bounds, content mask, or text style changed

• uncached: it is not embedded with .cached(...)

A hover inside an Entity::cached region should show that region notified and its siblings replaying. Anything else shows where a caching boundary leaks. get_frame_stats averages over the same frames. record_performance reports a fixed interval. In process, use Automation::mark_frames and Automation::frame_report.

bridge_ms is the work the bridge adds to a draw: finishing the accessibility tree, building the observed frame, and painting highlights. The semantic tree is converted off the UI thread when a client reads it.

Platform notes

• Linux: exact-window capture currently requires X11.

• Windows: screenshots take a short burst of compositor samples and return the newest, because Windows Graphics Capture can return a stale frame first. The one-second deadline covers waiting on the compositor, not readback, so even very large windows capture from debug builds.

Security

Only enable automation in development, testing, or another trusted environment. See SECURITY.md.

License

Apache-2.0.