Skip to main content
Glama
Sign In
dsh-computer-use

dsh-computer-use

by bluechonk
OverviewSchemaSourceRelated ServersDiscussionsAdmin
TypeScript
Local

dsh-computer-use

DeepSeek Harness 的 Computer Use MCP 服务器。通过 Accessibility 优先的语义操作驱动鼠标、键盘与界面元素，仅在 Accessibility 无法触达目标时回退到屏幕坐标。

功能

  • 观察 — 列出运行中的应用、读取 Accessibility 树、截取窗口/屏幕截图

  • 指针 — 点击、双击、右键、滚轮、拖拽、鼠标移动

  • 键盘 — 输入文本、按键组合、设置元素值

  • 控制 — 权限请求、剪贴板读写、显示器切换、会话停止

  • 验证 — 每次动作返回 action_receipt，state_id 驱动重新观察

Related MCP server: vision-input

平台

平台

Accessibility

输入

Windows

UIAccess (COM)

user32.dll (koffi)

macOS

AX API

CGEvent (koffi)

Linux

AT-SPI

XTest (koffi)

要求

  • Node.js >= 20

  • DSH >= 1.0.0（通过 dsh-mcp-client 消费）

  • Windows 10+ / macOS 13+ / Ubuntu 20.04+

快速开始

安装

git clone https://github.com/bluechonk/dsh-computer-use.git
cd dsh-computer-use
pnpm install
pnpm build

产物位于 dist/mcp/server.js。

配置 DSH

在 DSH profile 配置中添加：

- id: mcp-computer-use
  name: '@deepseek-ai/dsh-mcp-client'
  config:
    serverName: computerUse
    transport: stdio
    command: /path/to/dsh-computer-use/dist/mcp/server.js
    toolCallTimeoutMs: 90000

MCP 工具会以 mcp__computerUse__<tool> 格式出现在模型的工具列表中。

验证

pnpm build   # 类型检查 + 打包
pnpm smoke   # 验证 30 个工具注册
pnpm test    # 单元 + 集成测试

核心循环

观察 → 行动 → 验证
1. get_app_state     读取 Accessibility 树，获取 state_id
2. left_click        优先使用 element target（后台安全）
3. action_sent=true  确认动作已执行，重新观察

30 个工具

观察：list_apps · open_application · list_windows · get_app_state · screenshot · zoom · list_displays · switch_display · cursor_position

指针：left_click · double_click · triple_click · right_click · middle_click · scroll · left_click_drag · mouse_move · left_mouse_down · left_mouse_up

键盘：type · set_value · select_text · key · hold_key

语义：perform_action

运行时：request_access · stop_computer_control · wait · read_clipboard · write_clipboard

License

MIT

Available Tools

30 tools
cursor_positionA

Get the current cursor position in screen coordinates.

Inputs: no named parameters.

double_clickC

Double-click a target.

Inputs: target (required), strategy, modifiers, return_state.

get_app_stateB

Observe an app ONCE. Returns the accessibility tree + a fresh state_id that element targets reference. Call ONCE before acting; re-observe only after an action or when the next step needs fresh UI state.

Inputs: app_ref (required), detail.

hold_keyC

Hold a key for a duration (ms).

Inputs: text (required), duration (required), app_ref.

keyB

Press a non-text key or explicit chord (Return, Escape, cmd+a).

Inputs: text (required), app_ref, strategy.

left_clickC

Left-click a target. Element target is preferred (background-safe).

Inputs: target (required), strategy, modifiers, return_state.

left_click_dragC

Drag from one target to another. Both endpoints must resolve to the same app.

Inputs: to_target (required), from_target (required), modifiers.

left_mouse_downA

Press and hold the left mouse button at a target. Must be followed by left_mouse_up.

Inputs: target (required).

left_mouse_upC

Release a previously pressed left mouse button.

Inputs: target (required).

list_appsA

List running applications with their handles (pid, name, bundle_id, active).

Inputs: no named parameters.

list_displaysA

List all connected displays with their bounds and primary status.

Inputs: no named parameters.

list_windowsB

List an application's windows with bounds and focus info.

Inputs: app_ref (required).

middle_clickC

Middle-click a target.

Inputs: target (required), strategy, modifiers, return_state.

mouse_moveB

Move the virtual pointer to a coordinate target without clicking (hover).

Inputs: target (required).

open_applicationC

Launch or activate an application by name, pid, or bundle_id.

Inputs: pid, name, activate, bundle_id.

perform_actionC

Execute a named semantic action on an element (e.g. "press", "show_menu").

Inputs: action (required), target (required).

read_clipboardA

Read the current clipboard text content.

Inputs: no named parameters.

request_accessA

Request OS-level accessibility and screen recording permissions. Call once before acting.

Inputs: no named parameters.

right_clickC

Right-click a target to open its context menu.

Inputs: target (required), strategy, modifiers, return_state.

screenshotA

Capture a full-display screenshot. Prefer get_app_state. Use this only when accessibility cannot express the target and pixel coordinates are needed.

Inputs: app_ref, display_id.

scrollB

Scroll at a target by the given delta (positive = down/right).

Inputs: dx (required), dy (required), target (required), strategy.

select_textC

Select a text range within an editable element.

Inputs: end (required), start (required), target (required).

set_valueA

Atomically set the value of an editable element (preferred over type).

Inputs: value (required), target (required).

stop_computer_controlA

Immediately stop the active control session. Invalidates all pending actions.

Inputs: no named parameters.

switch_displayC

Switch the active display for screenshot and coordinate operations.

Inputs: index (required).

triple_clickC

Triple-click a target.

Inputs: target (required), strategy, modifiers, return_state.

typeC

Type text into an element or app-scoped target.

Inputs: text (required), target, strategy.

waitA

Wait for a specified duration (ms) before proceeding.

Inputs: duration (required).

write_clipboardB

Write text to the clipboard.

Inputs: text (required).

zoomC

Zoom into a region of the current raster for precise coordinate selection.

Inputs: target (required), region.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Deploy Server
    One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

    TDQS

    C2.9/5.0

    Scored across 30 tools

    Disambiguation4/5

    Most tools are distinct input/output primitives, but the fine-grained mouse controls (left_click vs left_mouse_down/up vs left_click_drag) and overlapping text-entry helpers (type, set_value, key, perform_action) can require careful reading. The descriptions clarify intended usage enough that an agent can mostly tell them apart.

    Naming Consistency4/5

    Names predominantly follow a snake_case verb_noun pattern such as list_apps, get_app_state, and read_clipboard, which makes the overall set predictable. A few bare verbs or nouns (wait, key, type, screenshot) and perform_action break the otherwise consistent rhythm.

    Tool Count2/5

    30 tools is on the high side and pushes past the typical well-scoped range. Many mouse variants and utility operations could be consolidated, making the tool surface feel heavier than necessary for an agent to navigate effectively.

    Completeness4/5

    The tool set covers the core computer-use lifecycle well: permissions, observation, mouse/keyboard input, clipboard, display/window inspection, and session control. Minor gaps such as window manipulation or app termination remain, but common automation tasks are achievable.

    Resources

    Related MCP Connectors

    Related MCP Servers