Serves as the primary LLM backend via any OpenAI-compatible chat completions endpoint (tested with DeepSeek), used to reason over the browser/DOM and Windows UIA observations and select actions.
English | 中文
Give your web agent access to Windows.
BridgeAgent is an AI agent that treats the browser and the Windows desktop as one workspace. It reads web pages through the DOM and native windows through Windows UI Automation, and switches between the two by a deterministic rule, not by the model. A task can start on a web page, continue in Explorer, Calculator, a dialog or any native app, and come back.
Status: alpha. Windows 11, Python 3.11+. Tested with DeepSeek through an OpenAI-compatible endpoint and with Claude through the Anthropic API.
Pain point today | What BridgeAgent does |
Browser agents (browser-use, Playwright MCP) stop at the edge of the page. A file picker, a Save dialog, a downloaded file that must be opened in a desktop app, and the agent is blind. | The same loop keeps going. The native window becomes the observation and the task continues there. |
Desktop agents (UFO2, computer-use) treat the browser as pixels or a flat control tree, so page content is read slowly, lossily and at screenshot prices. | Pages are read through the DOM and native windows through UI Automation, each with the observer that fits it. |
Handing the model two tool sets and letting it pick one leads to clicks on the wrong surface and keystrokes in the wrong window. | A program rule based on keyboard focus decides which observer and executor run. The model only picks the next action. |
Every repeat of the same task pays for the whole model loop again. | A successful run becomes a skill that replays without a model and is repaired by the model only when the UI changes. |
An agent that can reach the desktop can also pay, delete, send and read passwords out of the conversation. | A permission manifest, a native confirmation box for risky actions, passwords served from the OS keyring and an audit log. |
Related MCP server: Aether Browser MCP
One tree for two worlds. Page elements get b refs (Playwright), native controls get d refs (UI Automation).
The model sees one accessibility tree per step and picks one action.
Deterministic routing. focus.py looks at the foreground window's process and class and decides which observer
runs. Chrome dialogs drawn outside the page (print preview, permission bubbles) are found through UIA and merged in.
Focus-safe execution. If another app steals the foreground, the agent's window is brought back. Keystrokes are only sent when the observed window is still in front, and a vanished window makes the action fail instead of leak.
Skills without an LLM. A successful run is compacted into a skill. The next run replays it and spends one model call to verify. If the UI changed, the model takes over from the broken step and the skill is rewritten. Values copied from the page become variables, so a replay works on new data.
Guard rails the model cannot talk its way around. A permission manifest (allowed apps, folders, windows, credential sites), a native confirmation box for risky actions, passwords served from the OS keyring, and an audit log.
Three entry points, one core. CLI, a small desktop window, and an MCP server so other agents can use the same execution layer.
agent.py: observe -> route -> decide -> act, skills, self-heal
|
focus.py ----------- llm/ ------------- actions.py ----------- guard.py
who has focus? Anthropic / OpenAI one action schema, permissions,
compatible two executors confirmation, audit
| |
observers/browser.py observers/desktop.py BrowserExecutor DesktopExecutor
Playwright DOM scan Windows UIA walk (Playwright) (uiautomation)
\______________ tree.py: UINode / Observation ______________/
entry points: cli.py gui.py (PySide6) mcp_server.py
skills/: store, replay, params credentials/: keyring broker eval.py: task runnerRouting rule: the foreground window belongs to the agent's browser and uses the browser's main window class, so read the DOM. Anything else (a native window, a dialog owned by the browser, another browser, a Chrome popup) goes through UIA. Execution follows the ref prefix. The model never chooses the observer or the executor.
git clone https://github.com/Erikdai/bridge-agent.git
cd bridge-agent
python -m venv .venv
.\.venv\Scripts\pip install -e ".[gui,dev]"
.\.venv\Scripts\playwright install chromium
copy .env.example .env # add an API key (see below).env for DeepSeek (any OpenAI-compatible endpoint works the same way):
OPENAI_API_KEY=sk-...
OPENAI_BASE_URL=https://api.deepseek.com
OPENAI_MODEL=deepseek-chat
BRIDGE_PROVIDER=openaiRun a task that crosses the browser boundary:
bridgeagent run "Read the two numbers on the page, add them in the Windows Calculator app, report the result and close the calculator." --start-url file:///<repo>/tasks/pages/numbers.html
bridgeagent run "..." # the second run replays the learned skill with one verification call
bridgeagent gui # task box, progress, approvals, stop, result
bridgeagent mcp # execution layer over stdio for other agentsConnect it to Claude Code:
claude mcp add bridgeagent -- <repo>\.venv\Scripts\bridgeagent.exe mcpOther commands: focus, dump, scan URL (inspect trees without an LLM), cred set SITE (store a password in the OS
keyring), skill list|show|delete|replay, perms, eval (run tasks/examples.yaml), notify.
Safety: copy permissions.example.yaml to permissions.yaml to restrict which apps, folders, windows and credential
sites the agent may touch. Every action is appended to runs/audit.jsonl.
docs/EVAL.md: three passes over eight cross-boundary tasks with DeepSeek (first contact 8/8, pure replay for most
tasks at one model call), two Claude passes comparing tree-only with tree-plus-screenshot observation (7/8 and 8/8,
screenshots added cost but no fewer steps), a real Explorer-to-web upload workflow, and a ten-scenario deterministic matrix (scripts/bridge_matrix.py)
covering DOM-UIA-DOM transitions, Chrome in-window dialogs, focus hijacking before and after observation, refusal when the
target window is gone, LLM-free replay and 40 consecutive MCP calls. Small samples, not reliability statistics.
docs/RELATED_WORK.md: how this compares with UFO2, Agent S2, browser-use and RPA tools.
Windows only. Excel's cell grid exposes only visible header cells to UIA, so spreadsheet reading is not supported.
Chrome dialog detection relies on Chrome's view class names and may need updating across Chrome versions.
Skill matching is keyword overlap, and parameterization covers typed text and download paths, not file choices such as "the newest file".
Screenshot fallback for sparse trees needs a vision-capable model. BRIDGE_SCREENSHOTS=never|sparse|always picks the policy.
MIT. See LICENSE.
Click one element by ref: a page element through the DOM, or a native control through UI Automation.
Use for buttons, links, checkboxes, menu items, list items and tabs. Clicks on controls whose name suggests an
irreversible effect (Send, Pay, Delete, Submit, ...) first show the human a native confirmation box and are refused
if it is declined or times out. Returns a RESULT line (ok or ERROR with the reason) followed by a fresh observation,
so you do not need a separate observe call.
Inputs: ref (required). Returns: result
Fill a login form with a username and password that live in the operating-system keyring.
The secret is read by the executor and typed directly; it is never returned to you and never appears in the
observation. Use this instead of type_text for any password field. Works on web forms and on native login dialogs.
Returns a RESULT line naming the two fields that were filled, plus a fresh observation.
Inputs: site (required), ref, submit, password_ref. Returns: result
Bring the agent's browser window back to the front and return the page tree.
Use after working in a native app when the next step is on the web page again, or when observe reports that another application's window is in front.
Inputs: no named parameters. Returns: result
Start a Windows app, open a file or folder with its default app, or open a URI, and bring that window to the front.
This is how a task leaves the browser: after launch, observe shows the native window with d refs. Launching
something that is already open only re-focuses its window. Apps, folders and files outside the user's permission
manifest are refused. Returns a RESULT line and a fresh observation of the new window.
Inputs: command (required). Returns: result
List the full local paths of files the browser downloaded in this session, newest last.
Use the returned path with launch to open a downloaded file in its desktop app, or to tell the user where it is. Returns '(no downloads yet)' when nothing has been downloaded.
Inputs: no named parameters. Returns: result
List the skills learned from earlier successful runs: name, task description, step count and success record.
A skill is a recorded cross-environment procedure that replay_skill can execute without a model. Check this before
planning a task from scratch; if a listed description matches the task, replay it.
Inputs: no named parameters. Returns: result
Read the current screen as one accessibility tree and return it as text.
Call this first, and again whenever you need fresh refs. The result starts with a SURFACE line:
'browser' means the tree is the web page (refs b1, b2, ...); 'desktop' means it is the native window that has
keyboard focus such as Explorer, Calculator, a Save dialog or a Windows prompt (refs d1, d2, ...). A Chrome dialog
drawn on top of the page (print preview, permission bubble) is included with d refs. Each line shows the ref, role,
accessible name, value and state of an element, followed by a visible-text excerpt and the files downloaded so far.
Nothing is clicked or changed.
Inputs: start_url. Returns: result
Press a single key or key chord in the focused window.
Chords with Control, Alt or Windows modifiers and function keys are sent at operating-system level so they reach browser shortcuts and native apps; plain keys go to the page when the browser is in front. Pressing Enter while a risky control is focused asks the human first. Returns a RESULT line and a fresh observation.
Inputs: keys (required). Returns: result
Execute a learned skill step by step without a model and report what happened.
Each step re-finds its target by role and name on the current screen; values the skill learned from a page are re-read from the current page. The first line says whether all steps ran or at which step replay stopped and why, followed by a short trail of what was seen after each step and a fresh observation. If replay stopped, continue the task from the current state with the other tools.
Inputs: name (required), start_url. Returns: result
Scroll the page or the focused window to reveal more content, then return a fresh observation.
Use when the element you need is not in the tree yet; the browser observer only lists elements inside the viewport.
Inputs: ref, direction. Returns: result
Enter text into a field by ref, or send keystrokes to the focused window when no ref is given.
Keystrokes are only sent if the window observed last is still in front; if focus was stolen in between, the call
fails with an explanation instead of typing into the wrong window. Never use this for passwords: use
fill_credential, which keeps the secret out of the conversation. Returns a RESULT line and a fresh observation.
Inputs: text (required), ref, submit. Returns: result
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Scored across 12 tools
Each tool maps to a distinct GUI action (observe, click, type_text, press, scroll, navigate, launch, focus_browser, fill_credential, list_downloads, list_skills, replay_skill) with no overlapping responsibilities. The only potential overlap—observe, focus_browser, and navigate all return a page tree—is resolved by clear context triggers: read current screen vs. open a new URL vs. bring the browser forward.
All names use lowercase snake_case consistently, which prevents confusion. However, the set mixes single-verb imperatives (click, press, observe, scroll, navigate, launch) with verb_noun compounds (list_downloads, type_text, focus_browser, fill_credential, list_skills, replay_skill), so it is not a uniform verb_noun pattern throughout.
Twelve tools is well-scoped for a cross-environment GUI automation agent. Each tool earns its place by covering a necessary capability: observation, interaction, navigation, app launch, focus management, credential filling, downloads, and learned-skill replay.
The set covers the core GUI automation lifecycle well (observe, act, navigate, launch, focus, fill credentials, replay skills). Minor gaps remain, such as explicit wait-for-condition, tab/window management beyond simple focus, drag/hover, and clipboard read; agents can often work around these using key chords and fresh observations.
Control a real browser from AI agents: navigate, click, type, screenshot, read the page.
AI-powered web automation. Navigate websites using AI agents for one page or a thousand
AI-powered web automation. Navigate websites using AI agents for one page or a thousand
Autonomous cloud agent tasks: real browser + your tools, structured evidence-backed results.
