🖥️ screen-use

browser-use, but for the entire desktop.

Give any AI Agent eyes 👀 and hands 🖐️ on Windows — let Claude, Kimi, Cursor or your own agent see the screen, find UI elements, and operate any desktop app through natural language. No selectors. No scripts that break when the UI changes.

👆 Cross-app autonomy: the agent reads the result from Calculator, activates Notepad, and types it in — every step decided by the VLM watching the screen (see the live thought stream at the bottom).

Single-app agent loop (VLM computes 78 × 9 by itself):

Scripted cross-app (Calculator → Notepad):

Single-app precision clicking:

Why

Traditional RPA records selectors — and breaks the moment a page changes. browser-use (32k⭐) solved this for browsers. screen-use brings the same idea to the entire desktop: Excel, SAP clients, ERP software, even legacy Win32 programs.

Related MCP server: Windows-MCP

How it works

Your Agent (Claude / Kimi / Cursor / custom)   ← does the planning
        │  MCP or Python SDK
        ▼
┌─────────────────────────────────────────────┐
│ screen-use                                  │
│  Visual Loop ──► observe→think→act→verify   │
│  Introspection──► difficulty playbook        │
│  Meta-learning──► experience & vocab memory  │
│  Perception ──► UIA tree + screenshots (SoM)│
│  Locating   ──► strategy chain:             │
│                 ⓪ learned vocab mapping     │
│                 ① UIA text match (0 cost)   │
│                 ② Set-of-Mark + VLM         │
│  Action     ──► mouse / keyboard            │
└─────────────────────────────────────────────┘

VLM is optional, not required. The locating strategy chain hits most targets with pure Accessibility-tree text matching — zero model calls, millisecond latency. A vision model (cloud or local via Ollama) only kicks in for UIA-blind UIs.

Quickstart

git clone https://github.com/tongriyaotxt/screen-use.git
cd screen-use
pip install -r requirements.txt

As a Kimi CLI plugin (recommended)

One command — Kimi instantly gets all 14 desktop tools:

kimi mcp add --transport stdio screen-use -- <path-to-python.exe> -m screen_use.mcp_server
kimi mcp test screen-use   # verify the connection

Optionally install the bundled usage-strategy skill, which teaches Kimi the optimal tool-selection playbook:

mkdir -p ~/.kimi/skills/screen-use && cp skills/screen-use/SKILL.md ~/.kimi/skills/screen-use/

Then just tell Kimi: "Open Calculator and compute 123 × 456" or "Read what's in my Notepad".

As a generic MCP Server

Add to claude_desktop_config.json (or any MCP-compatible agent's config):

{
  "mcpServers": {
    "screen-use": {
      "command": "python",
      "args": ["-m", "screen_use.mcp_server"],
      "cwd": "path/to/screen-use"
    }
  }
}

Then just tell your agent: "Open Calculator and compute 123 × 456."

As a Python SDK

from screen_use import ScreenUse

tools = ScreenUse()

tools.click_element("Save")            # locate + click, one call
tools.type_text("Hello, 你好")          # Unicode-safe (clipboard paste)
tools.hotkey("ctrl", "s")

# Atomic tools for vision-capable agents:
elements = tools.list_ui_elements()    # id, name, type, bbox — no model needed
shot = tools.screenshot(annotate=True) # Set-of-Mark annotated screenshot
tools.click(500, 300)

Autonomous task loop

One call, full autonomy — the agent sees, decides, acts and self-corrects:

tools.run_task("打开计算器，算 25 乘以 4")   # observe → think → act → verify

Introspection (困难分类反思): when the loop gets stuck, it classifies the difficulty — no effect / repeat loop / consecutive failures / missing elements / unexpected popup — and reflects with a targeted prompt playbook, then adjusts strategy.

Meta-learning (元学习): successful runs are remembered. Similar past tasks are recalled as experience hints, and learned vocabulary mappings (e.g. "乘号" → Multiply by) become the strategy chain's new first level. It literally gets better the more you use it. Memory lives in ~/.screen_use/.

Tools (14)

Atomic (zero model dependency): screenshot · list_ui_elements · click · double_click · right_click · click_element_id · type_text · hotkey · press · scroll

High-level: find_element (strategy-chain locating) · click_element (locate + click) · read_screen (VLM screen Q&A) · run_task (autonomous visual loop)

Vision model (optional)

Only needed when your agent has no vision AND the target app is UIA-blind. Copy .env.example to .env:

Without any VLM configured, atomic tools and UIA matching still work fully.

Safety

• 🚨 Failsafe: slam your mouse to the top-left corner to abort instantly

• ✅ confirm_callback hook to approve every action (SDK)

• 🧪 ScreenUse(dry_run=True) records actions without executing

Extensibility

screen-use is designed as a set of replaceable layers — every tier can be extended without touching the core:

Safety hooks are part of the interface too: confirm_callback for human-in-the-loop approval, dry_run for simulation, PyAutoGUI failsafe for emergency stop.

Roadmap

• UIA + SoM locating strategy chain

• MCP Server (14 tools)

• Local VLM support (Ollama)

• Autonomous visual loop (run_task)

• Introspection playbook & meta-learning memory

• wait_for_element / auto-verification primitives

• Drag & drop

• VLM raw-coordinate fallback + OpenCV template matching (UIA-blind apps)

• macOS (Accessibility API) & Linux support

• PyPI release

Contributions welcome — see issues for good first tasks.

Development

pytest tests -q                        # 61 unit tests, no desktop/VLM needed
python examples/demo_calculator.py     # end-to-end demo (real clicks!)
python examples/mcp_client_demo.py     # MCP handshake + tool list

License

MIT

screen-use = 桌面版 browser-use：让任何 AI Agent 获得看屏幕、操作桌面应用的能力。

• 不是传统 RPA：不录制 selector，通过无障碍树 + 视觉模型理解 UI，界面变了也不怕

• 跨一切桌面应用：Excel、SAP、ERP 客户端、老旧 Win32 程序

• 自然语言驱动：click_element("保存按钮") 一句话搞定

• VLM 可选：策略链第一级是纯 UIA 文本匹配（零模型、毫秒级），视觉模型只在盲区兜底，支持本地 Ollama 保护隐私

接入方式、工具列表、安全配置与上文英文版一致。