UI Bridge MCP

MCP (Model Context Protocol) server for UI Bridge - enables AI to inspect and interact with UI elements.

Overview

This MCP server provides tools for:

• Inspecting UI elements - Get element positions, bounds, visibility, and state

• Interacting with elements - Click, type, focus, hover

• Two modes of operation:

  • Control mode: Interact with the qontinui-runner's own Tauri webview

  • SDK mode: Interact with external apps via the UI Bridge SDK

Related MCP server: computer-mcp

Installation

# Using pip
pip install ui-bridge-mcp

# Using poetry
poetry add ui-bridge-mcp

Prerequisites

The MCP server requires the qontinui-runner to be running, by default at 127.0.0.1:9876 (see Environment Variables).

For external app access, the target application must have the UI Bridge SDK integrated.

Configuration

Claude Desktop

Add to your claude_desktop_config.json:

{
  "mcpServers": {
    "ui-bridge": {
      "command": "ui-bridge-mcp"
    }
  }
}

Claude Code

Add to your MCP settings:

{
  "mcpServers": {
    "ui-bridge": {
      "command": "ui-bridge-mcp"
    }
  }
}

Environment Variables

Reaching a Windows runner from WSL

The runner binds 127.0.0.1 only, and its Host gate refuses, among others, a request whose Host is neither a loopback name with the runner's bound port nor a QONTINUI_RUNNER_ALLOWED_HOSTS entry (403 HOST_NOT_LOOPBACK).

• WSL1, or WSL2 in mirrored networking mode (networkingMode=mirrored in .wslconfig) shares the Windows loopback, so the default works. Every process in every WSL distro then reaches the runner as a loopback caller, credential routes included, so use it only where you trust every WSL user and container on the machine.

• WSL2 in NAT mode (the default) cannot reach the runner, because the runner listens on no address WSL can reach. Switch to mirrored mode. Forwarding a Windows port to the runner is not a supported setup: every client that can reach the forwarded address arrives at the runner as 127.0.0.1, with a local caller's access.

Available Tools

Health Check

Control Mode (Runner's Own UI)

SDK Mode (External Apps)

Note: The MCP server also includes legacy extension_* tools for browser tab access via a Chrome extension. These are deprecated and will be removed in a future release. Use the SDK tools instead.

Tool Results and Errors

A tool that could not do what was asked returns its result with isError: true, as the MCP spec requires. Branch on that flag, not on the text. This covers:

• the runner or app being unreachable, or answering with a failure (the text carries a [code=… status=…] tag when a code or HTTP status is known);

• a failure wrapped in a success envelope: a failed assertion (passed: false), a nested success: false, a wait that timed out, or an action that reports it did not run, change or delete anything;

• a missing or bad argument, an unknown tool, or an unmet precondition;

• an exception raised inside the tool, returned as Error: <exception> with no [code=…] tag.

When a response lacks the field that carries its verdict (an assertion's passed, an action's executed, sdk_status's connected, sdk_find's found, a design diff's hasSignificantChanges, and so on), the result says UNKNOWN and is flagged isError: true. It is never reported as a pass, a "no" or a "not found". Inside an answer that is otherwise complete, a missing detail is marked UNKNOWN in place without flagging the call, for example a form or field with no isValid/valid reads validity UNKNOWN, never valid.

An answer that happens to be empty is not an error and returns isError: false: an empty clipboard, zero matches, no saved bookmarks or baselines, or sdk_status reporting that no app is connected. sdk_status is an error only when the status request itself fails, and both texts begin "SDK not connected", so read the flag.

The flag needs mcp>=1.19.0, which pyproject.toml requires. Earlier releases replace the failure text with an SDK validation error.

Usage Examples

Inspect Runner UI

AI: Let me check what's on the runner's Settings page.

1. First, get a snapshot of the UI:
   ui_snapshot

2. Click the Settings button:
   ui_click element_id="sidebar-nav-item-settings"

3. Get another snapshot to see the new page:
   ui_snapshot

Inspect an SDK-Integrated App

AI: Let me check the login form on localhost:3000.

1. Connect to the app:
   sdk_connect url="http://localhost:3000"

2. Get all elements:
   sdk_elements

3. Type into the email field:
   sdk_type element_id="email-input" text="test@example.com"

4. Click the submit button:
   sdk_click element_id="login-button"

Element IDs

Elements in the runner's UI have registered IDs that follow patterns like:

• sidebar-nav-item-{name} - Sidebar navigation items

• button-{action} - Action buttons

• input-{field} - Input fields

• dialog-{name} - Dialog components

Use ui_snapshot to discover all available element IDs.

Architecture

Claude/AI
    |  MCP Protocol
ui-bridge-mcp (this server)
    |  HTTP
qontinui-runner (port 9876)
    |-- /ui-bridge/control/* --> Runner's Tauri webview
    |-- /ui-bridge/sdk/*     --> SDK-integrated apps (direct HTTP)

Development

# Install dependencies
poetry install

# Run tests
poetry run pytest

# Run linting
poetry run ruff check .
poetry run mypy --strict src tests

# Format code
poetry run ruff format src

License

Licensed under the MIT License. See LICENSE for full terms.