This MCP server bridges to UI-TARS desktop automation, letting you remotely control a desktop environment through five MCP tools:
screenshot — Capture the current screen as PNG or JPEG (default PNG).
click — Click at specified (x, y) screen coordinates.
type — Type text into the focused input field.
scroll — Scroll up/down/left/right by a given pixel amount (default 300).
browser_open — Open a URL in the default browser.
Provides tools for interacting with ByteDance's UI-TARS desktop automation client, enabling remote control of desktop environments via screenshot, click, type, scroll, and browser open operations through MCP.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@TarsUITarsBridge open https://github.com"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
MCP Bridge for UI-TARS — connects TSG gateway to UI-TARS desktop automation via Model Context Protocol.
你好！我是这个项目的作者，一名正在读小学的学生。 这个项目从规划到代码实现，完全由我自己主导设计，并借助 AI 工具完成。每一行代码、每一个功能模块，都凝聚了我对技术的热爱与坚持。 如果你认可这个项目，或者觉得它对你有帮助，希望能给我一个 Star 支持，这对我意义重大！
Related MCP server: computer-use
如果你在使用过程中遇到问题，欢迎提交 Issue 或参与讨论
如果你觉得项目不错，请点一个 Star，这是对我最大的鼓励
也欢迎分享给身边有需要的朋友，让更多人看到这个项目
后续功能迭代会继续推进，你的支持是我持续更新的动力
本项目依赖 AI 服务套餐，若因套餐额度耗尽或服务商调整，可能导致更新暂时停滞。 如遇较长时间未更新，请理解这是 AI 服务限制所致，并非项目放弃维护。 恢复服务后会第一时间继续推进，感谢你的耐心与包容。
TarsUITarsBridge wraps the UI-TARS desktop automation client as an MCP server with stdio transport, enabling TSG (TarsSecureGuard) gateway to remotely control desktop environments through a standardized protocol.
Independent repository — evolves separately from TSG gateway
Auto-include ready — TSG gateway can mount this bridge automatically via configuration
MCP-native — speaks standard Model Context Protocol (stdio transport in v0.1.0)
# Install dependencies
npm install
# Build
npm run build
# Run (mock mode if no UI-TARS endpoint configured)
npm startEnvironment Variable | Description | Default |
| UI-TARS API endpoint URL | (mock mode) |
| API key for the endpoint | — |
| Model name |
|
| Request timeout |
|
Tool | Description |
| Capture screen (PNG/JPEG) |
| Click at (x, y) coordinates |
| Type text into focused input |
| Scroll in direction (up/down/left/right) |
| Open a URL in the default browser |
v0.1.0 — MVP: stdio transport + 5 tools + mock endpoint ✅
v0.2.0 — HTTP transport dual-mode + official MCP server mount
v0.3.0 — TSG gateway auto-include protocol
v0.4.0 — Tauri sidecar integration (cross-platform)
v0.5.0 — Self-hosted vLLM deployment docs
v1.0.0 — 10+ tools, multi-device routing, penetration test report
In TSG gateway configuration, enable the bridge:
{
"bridges": {
"ui_tars": {
"enabled": true,
"command": "npx tars-ui-tars-bridge",
"env": {
"UI_TARS_ENDPOINT": "http://localhost:8080"
}
}
}
}Apache-2.0 — see LICENSE
Version bump with stability improvements
Open a URL in the default browser
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to open |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It does not say whether the call blocks until page load, whether it reuses or replaces an existing tab, what happens on navigation failure, or whether it establishes the session the sibling tools depend on.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One short, front-loaded sentence with no filler. It is efficient, though its brevity is partly the source of the missing behavioral context rather than a deliberate trade-off.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool that anchors an entire browser-automation suite, the description omits the session/state relationship to its siblings and gives no error or timing behavior. With no annotations or output schema to fall back on, the agent is left under-informed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and there is a single well-named required parameter, so the schema already documents everything. The description adds no format detail (e.g. scheme requirements) beyond what the schema provides, making the baseline 3 correct.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Open a URL in the default browser'), which is immediately distinguishable from the sibling page-interaction tools (click, type, scroll, screenshot). It does not explicitly name or contrast those siblings, so it falls short of a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance on when to use this versus alternatives, nor any prerequisite such as needing an open page before click/type/screenshot can operate. The agent must infer that this navigates a browser session from the tool name alone.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Click at the specified screen coordinates
| Name | Required | Description | Default |
|---|---|---|---|
| x | Yes | X coordinate | |
| y | Yes | Y coordinate |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden, yet it reveals nothing about behavior: whether this is a left/right click, whether it moves the pointer, whether it waits for navigation, or what happens on failure. For an input-simulating action with zero annotation coverage this is a meaningful gap.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single short sentence with no filler and the action front-loaded. It is efficient, though being only one line it leaves no room for the clarifying details noted above.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple two-parameter primitive with no output schema, the basics are covered. However, the coordinate space, click semantics, and success/failure signal are absent, leaving the agent unsure how to invoke it reliably in a browser context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Both parameters are documented in the schema at 100% coverage ('X coordinate', 'Y coordinate'), so the schema does the heavy lifting and 3 is the baseline. The description adds no extra meaning, such as the coordinate reference frame (screen vs. viewport) or units.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb ('Click') and the resource it acts on ('the specified screen coordinates'), which is enough to distinguish it from siblings like type, scroll, and screenshot. It stops short of explicitly contrasting itself with those siblings, so it does not reach the top band.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives no when-to-use or when-not-to-use guidance, no preconditions (e.g. window focus, element visibility), and never references alternatives such as type or scroll. The agent must infer usage entirely from the name.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Capture a screenshot of the current screen
| Name | Required | Description | Default |
|---|---|---|---|
| format | No | Image format (default: png) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, so the description carries the full burden. It implies a non-destructive read, but says nothing about what is actually captured (viewport vs. entire desktop), whether OS-level screen-recording permission is required, or how the image is returned. For a capture tool the scope and side-effect profile are the key unknowns and both are omitted.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One short, front-loaded sentence with no filler. It is appropriately sized for the operation, though its brevity is a symptom of under-specification rather than disciplined economy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, so the description should explain what comes back (raw image data, file path, URL), yet it does not. The ambiguous scope of 'current screen' combined with the silent return contract leaves real gaps for an agent invoking this tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and the single 'format' parameter is already documented with its enum and png default, so the schema does the heavy lifting. The description adds nothing about format semantics; baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb ('Capture') and resource ('screenshot') with a scope qualifier ('of the current screen'). It is trivially distinguishable from siblings (click, type, scroll, browser_open), though it never explicitly differentiates itself since no sibling competes for the same function.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this versus, e.g., browser_open or any other observation mechanism, and no prerequisites or conditions given. 'Current screen' implies a context but never states which context (browser viewport vs. full desktop).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Scroll the screen in the specified direction
| Name | Required | Description | Default |
|---|---|---|---|
| amount | No | Scroll amount in pixels (default: 300) | |
| direction | Yes | Scroll direction |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden and does not deliver. It says nothing about what is being scrolled (viewport vs. page), whether it waits for content, whether scrolling past the end is an error, or what the response contains.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single, front-loaded sentence with no filler. It is efficient, though arguably too terse for a tool with no annotations backing it up.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a two-parameter tool whose schema fully documents both parameters, the description is minimally sufficient. However, with no annotations and no output schema, the lack of any behavioral detail leaves gaps an agent may need to fill by trial and error.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so both parameters including the direction enum and the 300px default are already documented in the schema. The description only restates direction and adds no units, ranges, or edge-case behavior, so baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (scroll) and resource (the screen) with the scoping parameter (direction) named. It is clearly distinguishable from siblings like click, type, screenshot, and browser_open, though it does not explicitly say so.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No when-to-use guidance, no prerequisites, and no mention of alternatives or how scrolling interacts with the sibling interaction tools. The intent is inferable from the name but the description adds nothing about context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Type the given text into the focused input field
| Name | Required | Description | Default |
|---|---|---|---|
| text | Yes | Text to type |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full disclosure burden. It usefully reveals the prerequisite that an input must be focused, but omits key behavior: whether text is appended or replaces existing content, what happens when no field is focused, and whether typing is instant or simulated.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single, front-loaded sentence with zero padding; the essential action and target are stated immediately.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a one-parameter, no-output tool with full schema coverage, the description is nearly sufficient. Its only meaningful gap is edge-case behavior (nothing focused, append vs replace), which a more thorough definition would mention.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% for the single 'text' parameter ('Text to type'). The description adds no format, escaping, or special-character guidance beyond what the schema already provides, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (type) and target (the focused input field), which is clearly distinct from the browser siblings click, scroll, and screenshot. It does not explicitly name or contrast those alternatives, but the purpose is unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'focused input field' implies the field must already have focus (i.e., click first), which is useful implied guidance. However, there is no explicit statement of when to use this versus click/scroll, nor any note about what to do if nothing is focused.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0browser_openclickscreenshotscrolltypeScored across 5 tools
Each tool performs a clearly distinct action: screenshot, click, type, scroll, and browser_open. There is no overlap between their purposes.
Most tool names are single verbs (screenshot, click, type, scroll), which is consistent and readable. browser_open uses a noun_verb pattern, a minor deviation from the verb-centric convention.
The 5 tools are well-scoped for a UI automation bridge, covering essential interactions without excess.
Provides core UI automation functions (capture, click, type, scroll, open URL) but lacks operations like key presses, drag, or window management, which are common in such bridges.
Securely control computers you explicitly pair through files, terminals, processes, screenshots, desktop UI/input, clipboard, browser automation, diagnostics, and document tools.
Operate Linux, macOS and Windows from your LLM. Every action runs through an auditable allowlist.
Control a real browser from AI agents: navigate, click, type, screenshot, read the page.
Remote Desktop Commander: use files, terminal and processes on your own computers from any AI app.
