⚡ Flash computer use — Ultra-Fast Hybrid Browser & Desktop MCP Server

Flash computer use is a high-speed, precision-engineered Model Context Protocol (MCP) server in Python for controlling both web browsers (Playwright) and the physical OS desktop (Linux X11, Windows, macOS).

Unlike traditional screenshot-only computer-use agents that take 20–50 seconds per action and generate hallucinated [x, y] coordinates, Flash computer use combines instant deterministic DOM & native OS accessibility targeting (2–7 ms) with calibrated Set-of-Mark / hierarchical grid visual decision fallback (LiquidAI D1)—while always moving the real physical OS mouse pointer along a smooth, human-like minimum-jerk trajectory.

✨ Key Highlights

• 🚀 Up to 200× Faster Hybrid Execution & Deep DOM ↔ D1 Synergy (hybrid mode):

  • Context & Spatial DOM Disambiguation: Instantly resolves unique accessible labels, child SVG/image icon labels, <select> dropdown options, and even duplicate buttons across product cards or page zones (e.g. "Select Plan for Enterprise Suite" or "Sign in button in the header") in 2–7 ms (0 VLM calls), verifying unoccluded click coordinates via 9-point elementFromPoint hit-testing.

  • Smart Candidate Pre-Filtering & Cropping for D1: When visual disambiguation is needed on dense pages (60+ elements), select_d1_candidates narrows the Set-of-Mark annotation to only the relevant candidate subset, crops tightly around them, places non-occluding exterior badges, and enriches each option with [role | zone | card context].

  • Hybrid Offscreen Scrolling (scroll_until_visible): Detects offscreen DOM targets below the fold and scrolls directly until they enter the viewport without wasting intermediate VLM calls.

  • Desktop Native Accessibility: Queries native OS accessibility trees (Linux AT-SPI, Windows UI Automation, macOS AXUIElement) with automatic cache invalidation.

• 🖱️ Human-Like Physical OS Mouse Movement (Web + Desktop):

  • Always moves the real physical OS cursor along a minimum-jerk trajectory ($10t^3 - 15t^4 + 6t^5$) with a subtle natural wrist arc and exact pixel landing.

  • In visible browser mode, automatically maps Playwright viewport coordinates to global screen coordinates before every click, typing action, scroll, drag, or popup action.

• 🎨 Paint & <canvas> Drawing + Dragging:

  • Automatically detects the largest white/light drawing sheet (canvas_bounds) in desktop paint apps (MS Paint, KolourPaint, Pinta, etc.) or HTML5 <canvas> elements in the browser.

  • Supports smooth direct dragging (mouse_drag), continuous freehand & multi-stroke paths with anti-gap interpolation (draw_path), and 12 ready-to-use geometric shapes (draw_shape).

• 🛡️ Zero Coordinate Hallucination:

  • Uses LiquidAI D1 via native /v1/systemone decision probabilities (noul, choice, score with zero generated tokens) over Set-of-Mark badges and coarse-to-fine grids (Ollama, local Hugging Face Transformers, or Cloud API).

• 💻 Cross-Platform Precision:

  • Built-in Windows Per-Monitor DPI Awareness, macOS Retina multi-monitor scaling, and Linux X11 native clipboard & pointer integration.

Related MCP server: winagent

📖 Table of Contents

1. Speed & Accuracy Benchmarks

2. System Requirements

3. Step-by-Step Installation

4. Ollama & D1 Model Setup

5. MCP Client Configuration (Cursor, Claude, Codex, Antigravity)

6. Environment Variables Reference

7. Complete Reference of All 15 MCP Tools

8. Drawing in Paint, KolourPaint & Web Canvas

9. Cross-Platform Support (Linux, Windows, macOS)

1. Speed & Accuracy Benchmarks

Measured on a dense synthetic e-commerce page (50+ interactive elements) using LiquidAI D1 3B (Q8_0) on Ollama 0.40.4 (NVIDIA GTX 1050 4 GB VRAM). Full details and raw JSON outputs are in benchmarks/REPORT.md.

Why It Is So Fast

1. Deterministic DOM & Native Accessibility First (hybrid): Unique accessible labels and roles bypass VLM calls entirely (2–7 ms).

2. Candidate Region Cropping & Downscaling (vision): When D1 vision fallback is triggered, images are proportionally downscaled (D1_IMAGE_MAX_SIDE=768) and input candidates are tightly cropped (646×115), cutting GPU latency by 10× (26.8 s → 2.55 s) and increasing model confidence from 0.40 to 0.91.

3. Zero-Lag OS Pointer: Removes PyAutoGUI's default 100 ms per-step sleep, executing smooth minimum-jerk human curves in ~40 ms.

Reproduce the benchmarks locally:

# Hybrid mode benchmark (3 runs)
D1_BENCH_RUNS=3 .venv/bin/python benchmarks/browser_speed.py benchmarks/after.json

# Forced D1 Vision fallback benchmark
D1_BROWSER_STRATEGY=vision D1_BENCH_RUNS=1 .venv/bin/python benchmarks/browser_speed.py benchmarks/after-vision.json

# Desktop hierarchical grid precision benchmark
.venv/bin/python benchmarks/desktop_precision.py

2. System Requirements

• Python: >= 3.11

• Package Manager: uv (recommended) or pip

• Decision Engine Provider (choose one):

  • Ollama (recommended, local): version supporting /v1/systemone (e.g. >= 0.40) with d1:3b

  • Local Transformers: CUDA / Apple MPS / CPU with LiquidAI/d1-3B

  • Cloud API: any endpoint implementing /v1/systemone

• OS Desktop Prerequisites:

  • Linux: active X11 session (DISPLAY) and xclip or xsel for fast Unicode clipboard pasting.

  • Windows: interactive desktop session (Per-Monitor DPI Awareness is enabled automatically).

  • macOS: grant Accessibility and Screen Recording permissions to your terminal / MCP client.

3. Quick Start & Installation

Option A: Zero-Clone Install via npx (Recommended)

You can run or register Flash computer use directly from GitHub using npx—it automatically bootstraps an isolated Python virtual environment (~/.cache/flash-computer-use/venv or %LOCALAPPDATA%\flash-computer-use\venv) and installs Playwright Chromium on first launch:

# Run environment diagnostics via npx
npx -y github:shortsspectrumai-del/flash-computer-use doctor

# Generate portable MCP JSON config via npx
npx -y github:shortsspectrumai-del/flash-computer-use config --npx

# Start the stdio MCP server directly via npx
npx -y github:shortsspectrumai-del/flash-computer-use

Option B: Manual Clone & Local Virtual Environment

Linux / macOS

git clone https://github.com/shortsspectrumai-del/flash-computer-use.git
cd flash-computer-use

# 1. Create virtual environment and install dependencies
uv venv
uv pip install -e '.[dev]'

# 2. Install Playwright Chromium
PLAYWRIGHT_BROWSERS_PATH=.runtime/playwright .venv/bin/python -m playwright install chromium

Windows (PowerShell)

git clone https://github.com/shortsspectrumai-del/flash-computer-use.git
cd flash-computer-use

uv venv
uv pip install -e ".[dev,accessibility]"
$env:PLAYWRIGHT_BROWSERS_PATH=".runtime\playwright"
.\.venv\Scripts\python.exe -m playwright install chromium

4. Ollama & D1 Model Setup

Pull the d1:3b decision model and verify your environment:

# Pull LiquidAI D1 3B model
ollama pull d1:3b

# Check OS desktop capabilities, accessibility backend, and Python dependencies
npx -y github:shortsspectrumai-del/flash-computer-use doctor

# Verify D1 decision inference (binary noul, choice, score) on a synthetic image
D1_PROVIDER=ollama D1_MODEL_NAME=d1:3b npx -y github:shortsspectrumai-del/flash-computer-use check --inference

5. MCP Client Configuration (Cursor, Claude, Codex, Antigravity, Pi)

A. Cursor / Claude Desktop / Google Antigravity (mcp.json / claude_desktop_config.json)

Using npx (Works on Linux, macOS, and Windows without manual cloning)

Add the following block to your client's MCP configuration file (see examples/mcp.ollama.json):

{
  "mcpServers": {
    "Flash computer use": {
      "command": "npx",
      "args": [
        "-y",
        "github:shortsspectrumai-del/flash-computer-use"
      ],
      "env": {
        "D1_PROVIDER": "ollama",
        "D1_MODEL_NAME": "d1:3b",
        "D1_OLLAMA_BASE_URL": "http://localhost:11434",
        "D1_TIMEOUT_SECONDS": "120",
        "D1_BROWSER_STRATEGY": "hybrid",
        "D1_DESKTOP_STRATEGY": "hybrid",
        "D1_IMAGE_MAX_SIDE": "768",
        "D1_REAL_MOUSE": "true",
        "D1_MOUSE_DURATION": "0.04"
      }
    }
  }
}

(Or if using a local clone, set "command": "/absolute/path/to/flash-computer-use/run-mcp.sh" on Linux/macOS or "C:\\absolute\\path\\to\\flash-computer-use\\run-mcp.cmd" on Windows).

B. OpenAI Codex CLI (~/.codex/config.toml)

Register via npx with a single command:

codex mcp add "Flash computer use" \
  --env D1_PROVIDER=ollama \
  --env D1_MODEL_NAME=d1:3b \
  --env D1_OLLAMA_BASE_URL=http://localhost:11434 \
  --env D1_TIMEOUT_SECONDS=120 \
  --env D1_BROWSER_STRATEGY=hybrid \
  --env D1_DESKTOP_STRATEGY=hybrid \
  -- npx -y github:shortsspectrumai-del/flash-computer-use

Or copy examples/codex.ollama.toml into ~/.codex/config.toml:

[mcp_servers."Flash computer use"]
command = "npx"
args = ["-y", "github:shortsspectrumai-del/flash-computer-use"]
startup_timeout_sec = 60
tool_timeout_sec = 120

[mcp_servers."Flash computer use".env]
D1_PROVIDER = "ollama"
D1_MODEL_NAME = "d1:3b"
D1_OLLAMA_BASE_URL = "http://localhost:11434"
D1_TIMEOUT_SECONDS = "120"
D1_BROWSER_STRATEGY = "hybrid"
D1_DESKTOP_STRATEGY = "hybrid"
D1_IMAGE_MAX_SIDE = "768"
D1_REAL_MOUSE = "true"
D1_MOUSE_DURATION = "0.04"

C. Claude Code CLI

claude mcp add --transport stdio \
  --env D1_PROVIDER=ollama \
  --env D1_MODEL_NAME=d1:3b \
  --env D1_OLLAMA_BASE_URL=http://localhost:11434 \
  "Flash computer use" -- npx -y github:shortsspectrumai-del/flash-computer-use

D. Pi Coding Agent (~/.pi/agent/mcp.json)

pi mcp add flash-computer-use \
  --exposure direct \
  --env D1_PROVIDER=ollama \
  --env D1_MODEL_NAME=d1:3b \
  --env D1_OLLAMA_BASE_URL=http://localhost:11434 \
  --env D1_TIMEOUT_SECONDS=120 \
  --env D1_BROWSER_STRATEGY=hybrid \
  --env D1_DESKTOP_STRATEGY=hybrid \
  -- npx -y github:shortsspectrumai-del/flash-computer-use

6. Environment Variables Reference

See .env.example for a complete template.

7. Complete Reference of All 15 MCP Tools

Every tool call is serialized via an async lock and returns compact structured JSON including execution time (elapsed_ms).

Browser & Popup Navigation

1. browser_open(url): Opens a URL and returns page title, ready status, accessible input field labels, short text preview, and any detected popup/cookie banner details.

2. handle_popup(action="inspect" | "reject" | "accept" | "dismiss", max_scrolls=8): Inspects or resolves cookie banners, modal dialogs, and Google/YouTube consent pages (automatically scrolling terms if the action button is disabled until the bottom).

3. browser_scroll(direction="down" | "up", pixels=600, container="auto" | "popup" | "page"): Scrolls the active popup container or the main page.

Smart Hybrid Actions (mode="browser" or mode="desktop")

4. smart_click(target_description, mode="browser", double_click=False): Locates the target via DOM/accessibility or Set-of-Mark / hierarchical D1 grid, moves the real physical pointer along a human curve, and clicks an unoccluded point.

5. smart_move(target_description, mode="desktop"): Locates the target and moves the mouse pointer onto it without clicking (hover).

6. smart_type(target_description, text_to_type, press_enter=True, mode="browser"): Filters editable fields, replaces existing text, verifies the exact typed value (typed_verified=true), and optionally presses Enter.

7. scroll_until_visible(target_description, direction="down", max_scrolls=8, mode="browser"): Scrolls step-by-step until $P(\text{visible}) \ge 0.80$.

8. visual_assert(questions, mode="browser"): Evaluates 1–64 binary visual questions in a single D1 call. Probabilities between 0.20 and 0.80 return answer=null, certain=false.

9. wait_until_ready(timeout_seconds=10, mode="browser"): Fast DOM readiness check in hybrid mode or visual polling $P(\text{ready}) \ge 0.80$ in vision/desktop mode.

10. desktop_inspect(): Returns OS portability status, visible native accessibility elements, and auto-detected drawing sheet bounds (canvas_bounds).

Direct Mouse Control & Paint / Canvas Drawing

11. mouse_move(x, y, duration=0.04, mode="desktop"): Moves the pointer to (x, y) along a minimum-jerk human curve without clicking.

12. mouse_click(x=None, y=None, button="left", double_click=False, mode="desktop"): Clicks (left, right, middle) at the current cursor position or after moving to (x, y).

13. mouse_drag(start_x, start_y, end_x, end_y, button="left", duration=0.12, mode="desktop"): Moves to (start_x, start_y), presses the mouse button, drags smoothly along an interpolated path to (end_x, end_y), and safely releases.

14. draw_path(points, canvas_bounds=None, duration=0.4, mode="desktop"): Draws by dragging the mouse across the canvas sheet. Supports both a single polyline [[x, y], ...] and multi-stroke drawings [[[x, y], ...], ...] (lifting the mouse button between strokes). When canvas_bounds is omitted, automatically detects the active Paint sheet or <canvas>.

15. draw_shape(shape, x=None, y=None, width=None, height=None, canvas_bounds=None, duration=0.4, mode="desktop"): Draws a geometric shape (line, rectangle, ellipse, circle, triangle, diamond, star, arrow, heart, house, spiral, wave). Automatically detects the canvas and centers the shape when bounds or coordinates are omitted.

8. Drawing in Paint, KolourPaint & Web Canvas

Flash computer use can control raster and vector drawing applications just like a human user:

• Automatic Drawing Sheet Detection (canvas_bounds): You do not need to manually measure the canvas coordinates. The server inspects accessibility roles (canvas, drawing area) and falls back to a fast largest-rectangle histogram detector (detect_canvas_from_image) that locates the white/light drawing sheet inside Paint, KolourPaint, Pinta, or browser <canvas> apps.

• Anti-Gap Stroke Interpolation: Fast mouse drags in Paint often leave dotted gaps if points are far apart. interpolate_stroke subdivides long segments into dense ~6–8 px steps while preserving exact sharp corners.

• 12 Built-In Shapes (draw_shape): line, rectangle, ellipse, circle, triangle, diamond, star, arrow, heart, house, spiral, wave.

• Guaranteed mouseUp Safety: Every drag and drawing operation wraps pointer state in a try ... finally block so the mouse button is never left held down on error.

9. Cross-Platform Support (Linux, Windows, macOS)

🧪 Running the Test Suite

Run all 68 automated unit and integration tests (covering MCP stdio protocol, D1 HTTP contract, DOM & accessibility targeting, popup handling, human mouse trajectories, and real HTML5 <canvas> drawing):

.venv/bin/python -m pytest -q

📄 License

Released under the MIT License.