OpenUse

Production-grade dual-core agent framework for browser and native desktop automation.

Overview • Architecture • Benchmarks • Quickstart • MCP Integration • Python SDK • Documentation (中文)

Overview

Existing GUI agents face a fundamental trade-off:

• Web agents (e.g., standard browser drivers) cannot interact with native operating system applications like IM clients, office software, or system file pickers.

• Vision-based desktop agents (e.g., full-screen multimodal models) burn thousands of tokens per step, suffer from 5–15 second latencies per action, and drift on high-DPI displays.

OpenUse unifies web and native desktop automation through a dual-core architecture:

1. Zero-Vision Web Core: Uses direct Chrome DevTools Protocol (CDP) accessibility/DOM tree compaction. Decisions occur without streaming high-resolution screenshots.

2. Hardware-Accelerated Desktop Core: Runs local, offline neural perception (Apple Vision on Apple Silicon; RapidOCR ONNX on Windows) with sub-millisecond native input injection.

3. Sub-50ms Decision Loop: Replaces multi-second cloud LLM vision calls with TypeSafe Jev System One probabilistic action selection over structured Set-of-Marks (SoM).

Related MCP server: Hermes Browser MCP Server

Benchmarks

Measurements conducted across standard multi-step desktop and browser tasks (MacBook Pro M-Series / Windows 11 Core i7):

Architecture

graph TD
    subgraph Macro [Macro Orchestration]
        Goal[User Objective / MCP Request] --> Orchestrator[OpenAgent Engine Router]
    end

    subgraph CoreRouting [Execution Engines]
        Orchestrator -->|Web URLs & Web Applications| WebEngine[Browser Core - CDP]
        Orchestrator -->|Native Applications & OS Tasks| DesktopEngine[Desktop Core - HAL]
    end

    subgraph Perception [Local Perception Layer]
        WebEngine --> DOMTable[Compact Interactive DOM Table]
        DesktopEngine -->|macOS| MacHAL[Apple Vision Accurate Neural OCR + VNDetectRectangles]
        DesktopEngine -->|Windows| WinHAL[RapidOCR ONNXRuntime + Canny Contour Hierarchy]
    end

    subgraph DecisionEngine [System One Micro-Decision Loop]
        DOMTable & MacHAL & WinHAL --> NumericMap[Structured Control Index: btn_1 ... btn_n]
        NumericMap --> Jev[TypeSafe Jev System One Engine]
        Jev -->|sub-50ms Decision| ActionExecution[Native Hardware Event / CDP Action]
        ActionExecution --> Audit[Jev Judge State Verification]
    end

    Audit -.->|Success Confirmation| Goal

Hardware Abstraction Layer (HAL) Parity

Quickstart

Installation

git clone https://github.com/ribentianhuang38-boop/open-use.git
cd open-use

# Install core package
pip install -e .

# For Windows installations (includes ONNXRuntime, RapidOCR, and mss):
pip install -e ".[windows]"

Environment Configuration

Configure your TypeSafe Jev credentials:

cp .env.example .env
# Set JEV_API_KEY in .env

CLI Usage

# Desktop automation (automatically identifies OS platform)
openuse --mode desktop --goal "Open QQ and send /tmp/report.pdf to Project Team"

# Browser automation
openuse --mode browser --url "https://news.ycombinator.com" --goal "Find top 3 stories about compilers"

# Dual-core auto-dispatch
openuse --goal "Download latest metrics from https://internal.corp/dashboard and paste into Slack"

Model Context Protocol (MCP)

OpenUse implements standard JSON-RPC 2.0 Model Context Protocol over stdio, allowing immediate integration with Claude Desktop, Cursor, Windsurf, Zed, or Antigravity.

Server Launch

openuse --mcp

Claude Desktop Configuration

Add to claude_desktop_config.json:

{
  "mcpServers": {
    "open-use": {
      "command": "openuse",
      "args": ["--mcp"],
      "env": {
        "JEV_API_KEY": "your_typesafe_jev_key"
      }
    }
  }
}

Cursor Configuration

Add to .cursor/mcp.json:

{
  "mcpServers": {
    "open-use": {
      "command": "python",
      "args": ["-m", "open_use.mcp_server"],
      "env": {
        "JEV_API_KEY": "your_typesafe_jev_key"
      }
    }
  }
}

Manifest of Exposed Tools

Agent Skill & Deployment

OpenUse features a Trinity Integration Architecture to guarantee instant, zero-friction adoption across any agent platform:

Out-of-the-Box Deployment Guarantees

• macOS Zero-Setup: Includes precompiled, lightweight native binaries (ocr_detector and native_events). Zero Xcode or Swift toolchain installation needed. Automatically falls back to local JIT compilation or pure Python RapidOCR on edge environments.

• Windows Zero-Setup: 100% pure Python + Win32 ctypes (SendInput, CF_HDROP). Zero C++ or Visual Studio build tools required. ONNX runtime models are self-contained.

Python SDK

from open_use import OpenAgent

agent = OpenAgent()

# Chained cross-boundary task
agent.run_browser(
    url="https://github.com/ribentianhuang38-boop/open-use",
    goal="Star repository and copy release tag"
)

agent.run_desktop(
    goal="Open Notes application and paste the tag"
)

Contributing

Contributions are welcome. Please refer to CONTRIBUTING.md for development workflows, code standards, and PR submission guidelines.

License

This project is licensed under the Apache License 2.0. See the LICENSE file for details.

Acknowledgments

• browser-use: Architectural foundation for CDP browser interaction.

• RapidOCR: Embedded ONNX text recognition engine.

• TypeSafe AI: Jev System One sub-50ms probabilistic decision runtime.