Skip to main content
Glama
UI Bridge MCP
by qontinui

UI Bridge MCP

MCP (Model Context Protocol) server for UI Bridge - enables AI to inspect and interact with UI elements.

Overview

This MCP server provides tools for:

  • Inspecting UI elements - Get element positions, bounds, visibility, and state

  • Interacting with elements - Click, type, focus, hover

  • Two modes of operation:

    • Control mode: Interact with the qontinui-runner's own Tauri webview

    • SDK mode: Interact with external apps via the UI Bridge SDK

Related MCP server: computer-mcp

Installation

# Using pip
pip install ui-bridge-mcp

# Using poetry
poetry add ui-bridge-mcp

Prerequisites

The MCP server requires the qontinui-runner to be running, by default at 127.0.0.1:9876 (see Environment Variables).

For external app access, the target application must have the UI Bridge SDK integrated.

Configuration

Claude Desktop

Add to your claude_desktop_config.json:

{
  "mcpServers": {
    "ui-bridge": {
      "command": "ui-bridge-mcp"
    }
  }
}

Claude Code

Add to your MCP settings:

{
  "mcpServers": {
    "ui-bridge": {
      "command": "ui-bridge-mcp"
    }
  }
}

Environment Variables

Variable

Default

Description

QONTINUI_RUNNER_HOST

127.0.0.1

Runner hostname. The runner listens on the IPv4 loopback only; see Reaching a Windows runner from WSL.

QONTINUI_RUNNER_PORT

9876

Runner port. Empty or blank means the default; a value that is not a port (1-65535) is an error naming this variable.

Reaching a Windows runner from WSL

The runner binds 127.0.0.1 only, and its Host gate refuses, among others, a request whose Host is neither a loopback name with the runner's bound port nor a QONTINUI_RUNNER_ALLOWED_HOSTS entry (403 HOST_NOT_LOOPBACK).

  • WSL1, or WSL2 in mirrored networking mode (networkingMode=mirrored in .wslconfig) shares the Windows loopback, so the default works. Every process in every WSL distro then reaches the runner as a loopback caller, credential routes included, so use it only where you trust every WSL user and container on the machine.

  • WSL2 in NAT mode (the default) cannot reach the runner, because the runner listens on no address WSL can reach. Switch to mirrored mode. Forwarding a Windows port to the runner is not a supported setup: every client that can reach the forwarded address arrives at the runner as 127.0.0.1, with a local caller's access.

Available Tools

Health Check

Tool

Description

ui_health

Check if qontinui-runner is accessible

Control Mode (Runner's Own UI)

Tool

Description

ui_snapshot

Get complete UI state with all elements

ui_discover

Force element discovery/registration

ui_get_element

Get detailed info for a specific element

ui_click

Click an element by ID

ui_type

Type text into an input element

ui_focus

Focus an element

SDK Mode (External Apps)

Tool

Description

sdk_connect

Connect to an SDK-integrated app by URL

sdk_disconnect

Disconnect from the current app

sdk_status

Check SDK connection status

sdk_elements

Get all elements from the connected app

sdk_snapshot

Get a full UI snapshot from the connected app

sdk_click

Click an element by ID

sdk_type

Type into an element by ID

Journey Queries (the runner's observed navigation graph)

Each reads GET /apps/{app_id}/journey/* on the runner and takes app_id, plus the optional filter since / app_version / run_kind and a limit on the items listed per section (not journey_health). A from / to / node argument is a node key; journey_graph lists them and needs no starting key. The first line of every answer is COVERAGE: (ledger state, edges and nodes observed, open frontier, unmeasured dimensions, the filter the runner applied), so a truncated result still carries it. The second line is ANSWER: MEASURED, ANSWER: ABSENT (a settled negative) or ANSWER: UNKNOWN (<code>), which is never a negative; the affordances and departures that would settle it follow.

Tool

Description

journey_health

Whether the journey ledger is being written, and its counters

journey_graph

Observed node keys and edges, and the rows left out

journey_reachable_from

Nodes reachable from from, with step counts

journey_steps_to

Shortest observed path from from to to

journey_dead_ends

Settled dead ends (with back_only), separate from unexplored nodes

journey_requires_prior_knowledge

Nodes reached only by URL, never by an affordance

journey_frontier

Affordances seen and never activated, per node

journey_edge_timing

One edge's timing as a distribution (n, min/median/p90/max per metric)

Note: The MCP server also includes legacy extension_* tools for browser tab access via a Chrome extension. These are deprecated and will be removed in a future release. Use the SDK tools instead.

Tool Results and Errors

A tool that could not do what was asked returns its result with isError: true, as the MCP spec requires. Branch on that flag, not on the text. This covers:

  • the runner or app being unreachable, or answering with a failure (the text carries a [code=… status=…] tag when a code or HTTP status is known);

  • a failure wrapped in a success envelope: a failed assertion (passed: false), a nested success: false, a wait that timed out, or an action that reports it did not run, change or delete anything;

  • a missing or bad argument, an unknown tool, or an unmet precondition;

  • an exception raised inside the tool, returned as Error: <exception> with no [code=…] tag.

When a response lacks the field that carries its verdict (an assertion's passed, an action's executed, sdk_status's connected, sdk_find's found, a design diff's hasSignificantChanges, and so on), the result says UNKNOWN and is flagged isError: true. It is never reported as a pass, a "no" or a "not found". Inside an answer that is otherwise complete, a missing detail is marked UNKNOWN in place without flagging the call, for example a form or field with no isValid/valid reads validity UNKNOWN, never valid.

An answer that happens to be empty is not an error and returns isError: false: an empty clipboard, zero matches, no saved bookmarks or baselines, or sdk_status reporting that no app is connected. sdk_status is an error only when the status request itself fails, and both texts begin "SDK not connected", so read the flag.

The flag needs mcp>=1.19.0, which pyproject.toml requires. Earlier releases replace the failure text with an SDK validation error.

Usage Examples

Inspect Runner UI

AI: Let me check what's on the runner's Settings page.

1. First, get a snapshot of the UI:
   ui_snapshot

2. Click the Settings button:
   ui_click element_id="sidebar-nav-item-settings"

3. Get another snapshot to see the new page:
   ui_snapshot

Inspect an SDK-Integrated App

AI: Let me check the login form on localhost:3000.

1. Connect to the app:
   sdk_connect url="http://localhost:3000"

2. Get all elements:
   sdk_elements

3. Type into the email field:
   sdk_type element_id="email-input" text="test@example.com"

4. Click the submit button:
   sdk_click element_id="login-button"

Element IDs

Elements in the runner's UI have registered IDs that follow patterns like:

  • sidebar-nav-item-{name} - Sidebar navigation items

  • button-{action} - Action buttons

  • input-{field} - Input fields

  • dialog-{name} - Dialog components

Use ui_snapshot to discover all available element IDs.

Architecture

Claude/AI
    |  MCP Protocol
ui-bridge-mcp (this server)
    |  HTTP
qontinui-runner (port 9876)
    |-- /ui-bridge/control/* --> Runner's Tauri webview
    |-- /ui-bridge/sdk/*     --> SDK-integrated apps (direct HTTP)

Development

# Install dependencies
poetry install

# Run tests
poetry run pytest

# Run linting
poetry run ruff check .
poetry run mypy --strict src tests

# Format code
poetry run ruff format src

License

Licensed under the MIT License. See LICENSE for full terms.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • F
      license
      Not graded
      quality
      D
      maintenance
      Enables AI assistants to control Windows GUI by listing and focusing windows, capturing element snapshots via UIA/OCR/CDP, performing clicks/inputs/scrolls, verifying changes, waiting for screen updates, taking screenshots, and obtaining visual descriptions.
      2
      -
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables AI clients to inspect and control allowlisted Windows applications using screenshots, mouse actions, Unicode keyboard input, UI Automation, and an Esc/F12 emergency stop.
      1
      Apache 2.0
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables MCP-capable agents to locate and interact with Android and Windows GUI elements semantically—tapping, typing, waiting, and asserting—without hard-coded coordinates or model-based element localization.
      MIT