Serves as the primary LLM backend via any OpenAI-compatible chat completions endpoint (tested with DeepSeek), used to reason over the browser/DOM and Windows UIA observations and select actions.
English | 中文
Give your web agent access to Windows.
BridgeAgent is an AI agent that treats the browser and the Windows desktop as one workspace. It reads web pages through the DOM and native windows through Windows UI Automation, and switches between the two by a deterministic rule, not by the model. A task can start on a web page, continue in Explorer, Calculator, a dialog or any native app, and come back.
Status: alpha. Windows 11, Python 3.11+. Tested with DeepSeek through an OpenAI-compatible endpoint and with Claude through the Anthropic API.
Pain point today | What BridgeAgent does |
Browser agents (browser-use, Playwright MCP) stop at the edge of the page. A file picker, a Save dialog, a downloaded file that must be opened in a desktop app, and the agent is blind. | The same loop keeps going. The native window becomes the observation and the task continues there. |
Desktop agents (UFO2, computer-use) treat the browser as pixels or a flat control tree, so page content is read slowly, lossily and at screenshot prices. | Pages are read through the DOM and native windows through UI Automation, each with the observer that fits it. |
Handing the model two tool sets and letting it pick one leads to clicks on the wrong surface and keystrokes in the wrong window. | A program rule based on keyboard focus decides which observer and executor run. The model only picks the next action. |
Every repeat of the same task pays for the whole model loop again. | A successful run becomes a skill that replays without a model and is repaired by the model only when the UI changes. |
An agent that can reach the desktop can also pay, delete, send and read passwords out of the conversation. | A permission manifest, a native confirmation box for risky actions, passwords served from the OS keyring and an audit log. |
Related MCP server: Playwright MCP
One tree for two worlds. Page elements get b refs (Playwright), native controls get d refs (UI Automation).
The model sees one accessibility tree per step and picks one action.
Deterministic routing. focus.py looks at the foreground window's process and class and decides which observer
runs. Chrome dialogs drawn outside the page (print preview, permission bubbles) are found through UIA and merged in.
Focus-safe execution. If another app steals the foreground, the agent's window is brought back. Keystrokes are only sent when the observed window is still in front, and a vanished window makes the action fail instead of leak.
Skills without an LLM. A successful run is compacted into a skill. The next run replays it and spends one model call to verify. If the UI changed, the model takes over from the broken step and the skill is rewritten. Values copied from the page become variables, so a replay works on new data.
Guard rails the model cannot talk its way around. A permission manifest (allowed apps, folders, windows, credential sites), a native confirmation box for risky actions, passwords served from the OS keyring, and an audit log.
Three entry points, one core. CLI, a small desktop window, and an MCP server so other agents can use the same execution layer.
agent.py: observe -> route -> decide -> act, skills, self-heal
|
focus.py ----------- llm/ ------------- actions.py ----------- guard.py
who has focus? Anthropic / OpenAI one action schema, permissions,
compatible two executors confirmation, audit
| |
observers/browser.py observers/desktop.py BrowserExecutor DesktopExecutor
Playwright DOM scan Windows UIA walk (Playwright) (uiautomation)
\______________ tree.py: UINode / Observation ______________/
entry points: cli.py gui.py (PySide6) mcp_server.py
skills/: store, replay, params credentials/: keyring broker eval.py: task runnerRouting rule: the foreground window belongs to the agent's browser and uses the browser's main window class, so read the DOM. Anything else (a native window, a dialog owned by the browser, another browser, a Chrome popup) goes through UIA. Execution follows the ref prefix. The model never chooses the observer or the executor.
git clone https://github.com/Erikdai/bridge-agent.git
cd bridge-agent
python -m venv .venv
.\.venv\Scripts\pip install -e ".[gui,dev]"
.\.venv\Scripts\playwright install chromium
copy .env.example .env # add an API key (see below).env for DeepSeek (any OpenAI-compatible endpoint works the same way):
OPENAI_API_KEY=sk-...
OPENAI_BASE_URL=https://api.deepseek.com
OPENAI_MODEL=deepseek-chat
BRIDGE_PROVIDER=openaiRun a task that crosses the browser boundary:
bridgeagent run "Read the two numbers on the page, add them in the Windows Calculator app, report the result and close the calculator." --start-url file:///<repo>/tasks/pages/numbers.html
bridgeagent run "..." # the second run replays the learned skill with one verification call
bridgeagent gui # task box, progress, approvals, stop, result
bridgeagent mcp # execution layer over stdio for other agentsConnect it to Claude Code:
claude mcp add bridgeagent -- <repo>\.venv\Scripts\bridgeagent.exe mcpOther commands: focus, dump, scan URL (inspect trees without an LLM), cred set SITE (store a password in the OS
keyring), skill list|show|delete|replay, perms, eval (run tasks/examples.yaml), notify.
Safety: copy permissions.example.yaml to permissions.yaml to restrict which apps, folders, windows and credential
sites the agent may touch. Every action is appended to runs/audit.jsonl.
docs/EVAL.md: three passes over eight cross-boundary tasks with DeepSeek (first contact 8/8, pure replay for most
tasks at one model call), two Claude passes comparing tree-only with tree-plus-screenshot observation (7/8 and 8/8,
screenshots added cost but no fewer steps), a real Explorer-to-web upload workflow, and a ten-scenario deterministic matrix (scripts/bridge_matrix.py)
covering DOM-UIA-DOM transitions, Chrome in-window dialogs, focus hijacking before and after observation, refusal when the
target window is gone, LLM-free replay and 40 consecutive MCP calls. Small samples, not reliability statistics.
docs/RELATED_WORK.md: how this compares with UFO2, Agent S2, browser-use and RPA tools.
Windows only. Excel's cell grid exposes only visible header cells to UIA, so spreadsheet reading is not supported.
Chrome dialog detection relies on Chrome's view class names and may need updating across Chrome versions.
Skill matching is keyword overlap, and parameterization covers typed text and download paths, not file choices such as "the newest file".
Screenshot fallback for sparse trees needs a vision-capable model. BRIDGE_SCREENSHOTS=never|sparse|always picks the policy.
MIT. See LICENSE.
Click one element by ref: a page element through the DOM, or a native control through UI Automation.
Use for buttons, links, checkboxes, menu items, list items and tabs. Clicks on controls whose name suggests an
irreversible effect (Send, Pay, Delete, Submit, ...) first show the human a native confirmation box and are refused
if it is declined or times out. Returns a RESULT line (ok or ERROR with the reason) followed by a fresh observation,
so you do not need a separate observe call.
| Name | Required | Description | Default |
|---|---|---|---|
| ref | Yes | Element ref from the latest observation, e.g. 'b12' for a page element or 'd7' for a native window or dialog control. Refs are renumbered on every observation, so always use the most recent tree. |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden and does so well: it discloses a safety gate (native confirmation shown to the human for irreversible-sounding controls like Send, Pay, Delete, Submit) and the exact failure mode (refused if declined or timed out). It also states the return shape (RESULT line with ok or ERROR plus reason) and that a fresh observation follows, so no separate observe call is needed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three tight sentences: capability first, target classes second, safety and return behavior third. The parenthetical examples (Send, Pay, Delete, Submit) are illustrative rather than filler, and nothing is repeated unnecessarily.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a one-parameter action tool with an output schema available, the description covers everything an agent needs: what it acts on, which controls it suits, the confirmation/refusal policy, and that the observation is refreshed automatically. Missing details such as navigation waiting or disabled-element behavior are minor at this complexity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and the single ref parameter is already documented with the same examples and the renumbering caveat. The description adds no syntax, format, or semantic detail beyond what the schema provides, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (click), scoped to exactly one element by ref, and distinguishes the two backends it can act on: a page element through the DOM versus a native control through UI Automation. An agent can separate this from siblings like press, type_text or scroll without opening any schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives concrete context for use — buttons, links, checkboxes, menu items, list items and tabs — which tells the agent the intended target classes. It stops short of explicitly naming alternatives or stating when not to use it (e.g. keyboard activation via press, or typing into fields via type_text), so routing is implied rather than specified.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Fill a login form with a username and password that live in the operating-system keyring.
The secret is read by the executor and typed directly; it is never returned to you and never appears in the
observation. Use this instead of type_text for any password field. Works on web forms and on native login dialogs.
Returns a RESULT line naming the two fields that were filled, plus a fresh observation.
| Name | Required | Description | Default |
|---|---|---|---|
| ref | No | Ref of the username or email field. Optional: when empty, the field labelled user, email, login or account is used. | |
| site | Yes | Site key under which the user stored the credential with 'bridgeagent cred set <site>', e.g. 'company-portal'. Only keys allowed by the task or the permission manifest are accepted. | |
| submit | No | Press Enter in the password field afterwards to submit the login form. | |
| password_ref | No | Ref of the password field. Optional: when empty, the field labelled password is used. |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does so: it discloses that the secret is read by the executor, typed directly, never returned and never surfaced in the observation, and it describes the return shape (a RESULT line naming the two filled fields plus a fresh observation). That is exactly the security-relevant behavior an agent needs before touching a password field.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three tight sentences with no filler, and the secret-handling caveat and the type_text substitution are front-loaded where an agent will read them first. It is slightly longer than strictly required but each sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists and the description still summarizes the return, so return values are covered; secret handling, the alternative tool, and scope are all present. The only gap is failure behavior (missing keyring entry, disallowed site key), which is not addressed anywhere.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so ref, site, submit and password_ref are already fully documented in the schema, including defaults and the fallback labelling behavior. The description only says a username and password are filled, adding no syntax or constraint information beyond the schema, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (fill) and resource (login form) and an explicit source of truth (OS keyring credential), which no sibling does. It also names the sibling it replaces (type_text), so an agent can distinguish it without opening either schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives a clear routing rule: 'Use this instead of type_text for any password field,' plus the scope 'works on web forms and on native login dialogs.' It never states a when-not condition (e.g. what to do if the credential is absent from the keyring), so it stops short of full 5-level guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Bring the agent's browser window back to the front and return the page tree.
Use after working in a native app when the next step is on the web page again, or when observe reports that another application's window is in front.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full disclosure burden. It usefully reveals that the call re-focuses the OS window and returns the page tree, but says nothing about failure modes (e.g. no browser open), whether it disturbs the user's focus, or whether it is a pure read. Adequate but incomplete for a window/focus-mutating operation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences: the action is front-loaded, and the usage conditions follow. Every clause earns its place with no filler or restatement of the tool name.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so the return value needn't be explained, and the description still notes the page tree is returned. For a zero-parameter tool this is nearly complete; only edge-case behavior (no browser window, already focused) is left unaddressed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Zero parameters with 100% schema coverage, so there is nothing for the description to compensate for. Baseline 4 applies; no parameter semantics are needed or missing.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (bring the browser window to the front) plus the side-effect resource (return the page tree), which is unambiguous and easily distinguished from siblings like observe, click, or navigate. An agent knows exactly what this does without opening the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives two concrete triggers: after working in a native app when the next step is on the web, and when observe reports another application's window is in front. Strong contextual guidance, though it does not explicitly name an alternative tool or a when-not-to-use condition.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Start a Windows app, open a file or folder with its default app, or open a URI, and bring that window to the front.
This is how a task leaves the browser: after launch, observe shows the native window with d refs. Launching
something that is already open only re-focuses its window. Apps, folders and files outside the user's permission
manifest are refused. Returns a RESULT line and a fresh observation of the new window.
| Name | Required | Description | Default |
|---|---|---|---|
| command | Yes | What to open: a program name ('calc', 'notepad', 'excel'), a file or folder path ('C:\Users\me\Downloads\report.csv'), or a URI ('ms-settings:about'). Program names are resolved like the Windows Run box. |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided the description carries the full burden and does so well: it discloses the refocus-instead-of-relaunch behavior for already-open targets, the permission-manifest refusal boundary, and the return shape (a RESULT line plus a fresh observation of the new window). These are exactly the behavioral facts an agent needs before invoking.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the three openable target types, then the browser-exit context and behavioral caveats. Most sentences earn their place, though 'This is how a task leaves the browser' is slightly narrative and could be tightened.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite an output schema existing, the description still usefully sketches the return (RESULT line plus fresh observation), and it covers the permission boundary and refocus semantics for a one-parameter tool. Nothing needed to call it correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and the single 'command' parameter is already well documented (program name, path, or URI, resolved like the Windows Run box). The description restates the same three input categories without adding syntax, quoting, or escaping details beyond the schema, so the baseline of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (start/open) and enumerates the exact resource types handled: Windows apps, files, folders, and URIs, plus the side effect of foregrounding the window. It is clearly distinguishable from siblings like navigate (browser-level) and observe (used afterward to see the result).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives clear context for when this is the right call ('This is how a task leaves the browser') and routes the agent to observe for the follow-up step. It does not, however, contrast launch against navigate or focus_browser, which are the nearest alternatives for opening things, so the routing is helpful but not exhaustive.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
List the full local paths of files the browser downloaded in this session, newest last.
Use the returned path with launch to open a downloaded file in its desktop app, or to tell the user where it is. Returns '(no downloads yet)' when nothing has been downloaded.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the behavioral burden and does well: it discloses the session scope, the newest-last ordering, and the exact empty-state string '(no downloads yet)'. It omits any mention of permissions, platform constraints, or whether paths are guaranteed to still exist, which keeps it from a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three short sentences, front-loaded with the operation and its ordering, then usage, then the empty-state edge case. Every sentence earns its place with no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter list tool with an output schema already defining return structure, the description supplies the remaining necessary context: scope, ordering, and the empty-state sentinel. Nothing an agent needs to call it correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, so the baseline is 4; the description has no parameters to annotate and correctly doesn't invent any. No additional parameter meaning is required or missing.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (List) and resource (local paths of downloaded files), plus scope ('in this session') and ordering ('newest last'). This clearly distinguishes it from siblings like observe or launch, which deal with browser interaction rather than enumerating downloads.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly routes the agent to the sibling tool 'launch' for opening a downloaded file or telling the user its location, which is genuine usage guidance. It stops short of stating when not to use this tool or preconditions, but the forward-reference to launch is concrete.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
List the skills learned from earlier successful runs: name, task description, step count and success record.
A skill is a recorded cross-environment procedure that replay_skill can execute without a model. Check this before
planning a task from scratch; if a listed description matches the task, replay it.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the behavioral burden and does well: it defines a skill as a recorded cross-environment procedure that replay_skill can execute without a model, which is real context about the data's origin and semantics. It is a read-only listing so safety disclosure is trivial, but it says nothing about ordering, scope limits, or whether prior runs of other environments are included.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with what is returned, followed by the definition of a skill and the routing advice; every sentence is doing work. The phrase 'learned from earlier successful runs' is echoed by 'success record' in the same sentence, a minor redundancy, but there is no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists so return-value detail is not required, and there are no parameters or annotations to compensate for. What remains — the meaning of a skill, the relationship to replay_skill, and when to call this first — is all present, so an agent has everything needed to call it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Zero parameters, so there is no per-parameter semantics to explain; the baseline for a parameterless tool applies. Schema coverage is 100% and the description adds no syntax that could conflict or confuse.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('List the skills learned from earlier successful runs') and enumerates the returned fields (name, task description, step count, success record). It also positions the tool against the sibling replay_skill by explaining that replay_skill executes what this lists, so an agent can tell the two apart without opening either schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives an explicit when-to-use ('Check this before planning a task from scratch') and a concrete follow-up condition ('if a listed description matches the task, replay it'), naming the alternative tool. Nothing about tool selection is left to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Read the current screen as one accessibility tree and return it as text.
Call this first, and again whenever you need fresh refs. The result starts with a SURFACE line:
'browser' means the tree is the web page (refs b1, b2, ...); 'desktop' means it is the native window that has
keyboard focus such as Explorer, Calculator, a Save dialog or a Windows prompt (refs d1, d2, ...). A Chrome dialog
drawn on top of the page (print preview, permission bubble) is included with d refs. Each line shows the ref, role,
accessible name, value and state of an element, followed by a visible-text excerpt and the files downloaded so far.
Nothing is clicked or changed.
| Name | Required | Description | Default |
|---|---|---|---|
| start_url | No | Optional URL to open in the agent's browser before observing, e.g. 'https://example.com' or a file:// page. Leave empty to observe the current state. |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations the description carries the full burden, and it does well: 'Nothing is clicked or changed' declares the read-only, side-effect-free profile, and it explains result structure, SURFACE semantics, ref prefixes (b/d), and embedded content like downloaded files. It omits auth constraints and any rate-limit/cost behavior, so it is strong but not exhaustive.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the core action, then dense but relevant detail on refs and the SURFACE line. It is on the long side and could tighten the mid-section, but each sentence conveys operational meaning rather than filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists so return values need not be re-explained, yet the description usefully clarifies the semantics of refs that schema alone may not convey. The main gap is the undefined boundary with navigate/launch for opening a URL via start_url, which an agent must infer.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and the single optional start_url is documented in the schema with a default and example. The description adds nothing about start_url and doesn't clarify its relationship to the navigate sibling, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Read the current screen as one accessibility tree and return it as text') and defines the exact return shape. An agent can distinguish this observation tool from the action siblings (click, type_text, press) without opening any schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
'Call this first, and again whenever you need fresh refs' gives clear timing guidance tied to a concrete condition. It doesn't name when not to use it or route to an alternative (e.g. navigate/launch for opening pages), so it stops short of full when/when-not/alternatives coverage.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Press a single key or key chord in the focused window.
Chords with Control, Alt or Windows modifiers and function keys are sent at operating-system level so they reach browser shortcuts and native apps; plain keys go to the page when the browser is in front. Pressing Enter while a risky control is focused asks the human first. Returns a RESULT line and a fresh observation.
| Name | Required | Description | Default |
|---|---|---|---|
| keys | Yes | One key or chord in Playwright syntax: 'Enter', 'Escape', 'Tab', 'F2', 'Control+S', 'Alt+F4', 'Control+Shift+S'. Not a string of characters; use type_text for that. |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does so well: it discloses that modified/function keys are dispatched at OS level while plain keys target the page, and that Enter on a risky control prompts the human first. It also notes the return shape (RESULT line plus fresh observation), which is unusually rich disclosure.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two short paragraphs, front-loaded with the core action before the behavioral detail. The second sentence is long but every clause adds non-redundant information; no padding or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists so return values need no elaboration, and the unannotated safety profile is covered by the human-confirmation note for risky Enter presses. For a one-parameter tool, nothing an agent needs to call it correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the Playwright syntax and examples are already fully documented in the property. The description adds only the loose 'single key or key chord' framing, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource — 'Press a single key or key chord in the focused window' — with a scoping detail (single key vs. character string) that implicitly separates it from type_text. It does not explicitly name type_text in the description text, so sibling differentiation is present but not spelled out verbatim.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explains the routing behavior of modifier chords vs. plain keys, which implies usage, but never states when to choose press over type_text, click, or other siblings. The closest guidance ('use type_text for that') lives in the schema parameter, not the description.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Execute a learned skill step by step without a model and report what happened.
Each step re-finds its target by role and name on the current screen; values the skill learned from a page are re-read from the current page. The first line says whether all steps ran or at which step replay stopped and why, followed by a short trail of what was seen after each step and a fresh observation. If replay stopped, continue the task from the current state with the other tools.
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | Skill name exactly as printed by list_skills. | |
| start_url | No | Optional URL to open before replaying, when the skill starts on a web page. |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does substantial work: it discloses that no model is needed, that targets are re-found by role/name on the current screen, and that learned values are re-read from the current page rather than cached. It also describes the output shape and failure behavior. It omits side-effect/permission context, but core runtime behavior is well covered.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three tight sentences, front-loaded with the purpose, then mechanism, then output/failure handling. Nothing is filler and each sentence carries distinct information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return values need not be explained, yet the description still summarizes the report format helpfully. Combined with mechanism and failure routing, it is nearly complete; only side-effect implications of executing a skill are left implicit.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so both parameters are already documented with their formats. The description adds behavioral context about re-finding targets and re-reading values, which informs how replay uses the skill, but does not extend the meaning of either parameter beyond the schema. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource ('Execute a learned skill step by step without a model') and adds the distinguishing scope that it runs without a model and reports results. This clearly separates it from the atomic action siblings like click and type_text.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides explicit post-failure routing: 'If replay stopped, continue the task from the current state with the other tools.' This tells the agent what to do next when this tool ends abnormally. It does not, however, say when to prefer replay over manually re-issuing atomic steps, so no full alternative-selection guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Scroll the page or the focused window to reveal more content, then return a fresh observation.
Use when the element you need is not in the tree yet; the browser observer only lists elements inside the viewport.
| Name | Required | Description | Default |
|---|---|---|---|
| ref | No | Optional ref of a scrollable element to scroll over; defaults to the whole page or window. | |
| direction | No | 'down' or 'up'. | down |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full behavioral burden. It discloses a key trait worth knowing: scrolling returns a fresh observation and the observer only surfaces in-viewport elements. However it omits scroll magnitude, direction defaults, behavior at the end of the page, and any focus prerequisite for the 'focused window' case.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with no filler. The action and its side effect come first, then the when-to-use condition, so the agent gets purpose before caveat.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return-value detail is unnecessary, and the description appropriately references the fresh observation instead. For a zero-required-parameter tool it is nearly complete; it could still say what happens at the page boundary or whether scrolling is incremental.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and the parameters are optional with defaults already documented, so the schema does the heavy lifting. The description echoes the 'whole page or focused window' default but adds no syntax or semantics beyond what the schema states.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (scroll), the target (page or focused window), and the post-condition (reveal more content, return a fresh observation). It also implicitly separates itself from the observer by explaining the observer only lists viewport elements, so an agent can route between scroll and observe without opening either schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives an explicit trigger: use when the element you need is not in the tree yet, because the observer only lists elements inside the viewport. Clear context for adoption, but it names no exclusion and does not explicitly point to a sibling tool (e.g. observe) as the alternative to skip.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Enter text into a field by ref, or send keystrokes to the focused window when no ref is given.
Keystrokes are only sent if the window observed last is still in front; if focus was stolen in between, the call
fails with an explanation instead of typing into the wrong window. Never use this for passwords: use
fill_credential, which keeps the secret out of the conversation. Returns a RESULT line and a fresh observation.
| Name | Required | Description | Default |
|---|---|---|---|
| ref | No | Ref of the text field to fill (it is cleared first). Leave empty to send the keystrokes to whatever currently has keyboard focus, which is right for calculators, address bars and dialogs that already have focus. | |
| text | Yes | The exact text to enter, e.g. 'report-2026.csv' or a whole calculator expression such as '1234+5678='. | |
| submit | No | Press Enter after typing, e.g. to submit a search box or confirm a file name. |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does well: it discloses the focus-guard failure mode (call fails rather than typing into the wrong window), the credential safety constraint, and the return shape (RESULT line plus fresh observation). It stops short of stating whether the field is cleared on failure or what happens on partial input, but the safety-critical behavior is covered.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the core action and its two modes, and the safety warning is placed before the return-value note. The first sentence largely restates what the schema parameters already say, which is the only redundancy; the middle sentences each earn their place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a three-parameter interaction tool with an output schema, the description covers purpose, mode selection, failure semantics and the security exclusion. Since a return schema exists, it need not detail the observation contents, and nothing an agent needs to call this correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so all three parameters (ref, text, submit) are already documented with examples. The description's mention of ref-vs-no-ref behavior duplicates the schema rather than adding syntax, edge-case or precedence detail, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Enter text into a field by ref, or send keystrokes to the focused window') and immediately distinguishes the two operating modes. It also names the sibling it must not be confused with (fill_credential), so an agent can route correctly without opening a schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicit when-to-use for both modes: ref when targeting a specific field, no ref for calculators, address bars and dialogs that already hold focus. It adds an explicit when-not ('Never use this for passwords') plus the alternative tool to use instead.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0clickfill_credentialfocus_browserlaunchlist_downloadslist_skillsnavigateobservepressreplay_skillscrolltype_textScored across 12 tools
Each tool maps to a distinct GUI action (observe, click, type_text, press, scroll, navigate, launch, focus_browser, fill_credential, list_downloads, list_skills, replay_skill) with no overlapping responsibilities. The only potential overlap—observe, focus_browser, and navigate all return a page tree—is resolved by clear context triggers: read current screen vs. open a new URL vs. bring the browser forward.
All names use lowercase snake_case consistently, which prevents confusion. However, the set mixes single-verb imperatives (click, press, observe, scroll, navigate, launch) with verb_noun compounds (list_downloads, type_text, focus_browser, fill_credential, list_skills, replay_skill), so it is not a uniform verb_noun pattern throughout.
Twelve tools is well-scoped for a cross-environment GUI automation agent. Each tool earns its place by covering a necessary capability: observation, interaction, navigation, app launch, focus management, credential filling, downloads, and learned-skill replay.
The set covers the core GUI automation lifecycle well (observe, act, navigate, launch, focus, fill credentials, replay skills). Minor gaps remain, such as explicit wait-for-condition, tab/window management beyond simple focus, drag/hover, and clipboard read; agents can often work around these using key chords and fresh observations.
AI-powered web automation. Navigate websites using AI agents for one page or a thousand
AI-powered web automation. Navigate websites using AI agents for one page or a thousand
Autonomous cloud agent tasks: real browser + your tools, structured evidence-backed results.
Browser-backed QA with evidence and fix-ready reports for coding agents.
