Skip to main content
Glama
Sign In
grok-computer-mcp

grok-computer-mcp

by bo-516
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Local

Grok Computer

English | 简体中文

Computer use for Grok Build: a Grok Build plugin (computer-use) plus a purpose-built MCP server (the grok-computer-mcp facade) that let Grok Build see the screen, click and type in desktop apps and browsers on macOS, Windows and Linux. The main use is checking UI inside the development loop (change code → launch the app → look at the UI → change again), and operating desktop and web tools that have no API.

Design document: docs/goal.md. Installation and configuration: docs/install.md. The documents under docs/ are written in Chinese.

How it works

main agent ──delegates──▶ computer subagent (own context; only reports go back)
                            │
                            ├─▶ computer: grok-computer-mcp facade ──▶ Cua Driver ──▶ desktop (host / sandbox)
                            └─▶ browser: Playwright MCP (--isolated)
hooks: guard (risk grading and confirmation) · audit · verify_gate (report and final check) · selfcheck

  • Screenshots stay out of the main session: all GUI work goes to the computer-use:computer subagent; the main agent only receives a structured report of at most 2 KB (STATUS / SUMMARY / EVIDENCE / BLOCKERS / NEXT).

  • Accessibility tree first, pixels last: an observation is a compact list of elements by default (one interactive element per line, with a stable ref). When the tree is not enough it falls back to a numbered Set-of-Mark screenshot, and only then to a grounding model or raw coordinates.

  • One coordinate space: every coordinate the model sees or sends is a pixel of the current observation image (long edge 1280 px). HiDPI scaling, window offsets and multiple displays are all handled inside the facade, and an action based on an observation of a screen that has since changed is refused (STALE_OBSERVATION).

  • Two safety layers: the hooks ask you before high-risk actions (send, delete, pay, ...), and the facade enforces hard rules itself and refuses everything when its policy cannot be loaded (deny-listed apps, secure fields, dangerous key chords, a session lock, an emergency stop).

The facade exposes nine tools: observe, click, type_text, press_keys, scroll, drag, apps, wait_for and locate.

Related MCP server: Open Computer Use

Quick start

curl -fsSL https://cua.ai/driver/install.sh | bash      # desktop driver (macOS also needs permissions, see the install guide)
curl -LsSf https://astral.sh/uv/install.sh | sh         # runs the facade
grok plugin marketplace add bo-516/computer-use
grok plugin install computer-use --trust
grok plugin enable computer-use
uvx grok-computer-mcp@0.2.0 doctor                      # self-check

Then describe the task to grok in plain language, or use /computer open Settings, turn on dark mode and confirm it took effect.

Privacy: screenshots are sent to the model provider. Deny-listed apps (password managers, keychains, terminals, banking apps, ...) are never observed; traces stay on your machine and are deleted after 7 days by default. See section 6 of the install guide.

Repository layout

Path

Contents

.grok-plugin/

Marketplace index and component catalog

plugins/computer-use/

The plugin: subagent, skill and platform notes, /computer command, hooks, .mcp.json

servers/grok-computer-mcp/

The facade MCP server (Python 3.11+, published on PyPI, run with uvx)

eval/

Task set (60 Cua Bench variants), host-mode runner and metrics, grounding benchmark, Phase 0 probes; see eval/README.md

docs/

Design document, install guide, Phase 0 verification record, state file schema

scripts/

hooklib sync, plugin index generator

tests/

Hook fixture tests and repository contract tests

Development

Requires uv (it installs Python 3.11 when needed); the dev-loop task tests also need Node.js.

uv sync
uv run pytest                 # all tests (fake backend; no desktop or account needed)
uv run ruff check
uv run pyright
uv run --python 3.8 --no-project --with pytest==8.3.5 pytest tests/hooks   # hooks on Python 3.8
uv run python scripts/sync_hooklib.py --check     # the facade's hooklib copy matches the plugin
uv run python scripts/plugin_index.py --check     # the component catalog is current
grok plugin validate plugins/computer-use

Conventions are in AGENTS.md. The plugin directory ships files only and the hooks use the standard library only; the facade is the only code that talks to Cua Driver; the safety rules (fail-closed facade, R3 never loosened, typed text redacted, ...) must not regress.

Status

The plugin, the facade, the task set, the grounding benchmark and the Phase 0 probe kit are implemented and tested against a fake backend and a fake Cua Driver. What still needs a real environment (running the Phase 0 probes and the grounding benchmark, running the task set on macOS, Windows and Linux machines) is listed in docs/goal.md §10.1 and docs/phase0-verification.md.

License

Apache-2.0.

Available Tools

9 tools
appsApps and windowsB
Idempotent

list apps, list windows (optionally of one app), launch an app in the background, or focus (bring to front) an app.

Inputs: action (required), name, observation_id. Returns: object

clickClickA
Destructive

Click an element: prefer ref, then mark (from a som observation), then point (image pixels, last resort). Pass the observation_id you act on. Returns what changed and the next observation_id.

Inputs: observation_id (required), ref, mark, count, +3 more. Returns: object

dragDragA
Destructive

Drag from one ref/mark/point to another in the same observation.

Inputs: to (required), from (required), observation_id (required). Returns: object

locateLocate by descriptionA
Read-onlyIdempotent

Find a described target in the current screenshot with a grounding model when refs and marks fail. Returns candidate points in image pixels; when not confident, up to 3 candidates and an annotated screenshot.

Inputs: description (required), observation_id (required). Returns: object

observeObserve the screenA
Read-onlyIdempotent

Observe the GUI: interactive elements with refs (e7) and [x,y,w,h] boxes in image pixels. mode auto returns the tree, or a Set-of-Mark screenshot when the tree is thin; screenshot/som attach an image. root_ref expands a container; scope='screen' lists windows for cross-app work. Start every step with this.

Inputs: app, mode, page, scope, +3 more. Returns: object

press_keysPress keysA
Destructive

Press a key chord such as 'cmd+s', 'enter' or 'tab', or a list of chords in order. Quit, log-out, lock, task-manager and run-dialog shortcuts are refused.

Inputs: keys (required), observation_id (required). Returns: object

scrollScrollB

Scroll the window, or at a ref/mark/point, by wheel notches.

Inputs: direction (required), observation_id (required), ref, mark, +2 more. Returns: object

type_textType textA
Destructive

Type text into the field ref, or the focused field. Refused for password and credential fields: never type secrets. clear_first empties the field first.

Inputs: text (required), observation_id (required), ref, clear_first. Returns: object

wait_forWait for an elementA
Read-onlyIdempotent

Wait until an element whose label or value contains text (optionally of a role) appears, or disappears with gone=true. Returns a fresh observation.

Inputs: app, gone, text, ref_role, +2 more. Returns: object

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Deploy Server
    One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

    TDQS

    A3.8/5.0

    Scored across 9 tools

    Disambiguation5/5

    Each tool targets a clearly distinct action: observe for perception, wait_for for synchronization, locate for model-based grounding, and click/type_text/press_keys/scroll/drag for distinct input primitives. The observe/wait_for/locate trio is well-differentiated by intent and fallback role, so an agent can reliably pick the right one.

    Naming Consistency4/5

    Mostly consistent lower_snake_case, with verb_noun forms (wait_for, type_text, press_keys) and clean single verbs (observe, click, scroll, drag, locate). The lone noun-style name 'apps' deviates slightly, but it is readable and unambiguous.

    Tool Count5/5

    Nine tools is well-scoped for a GUI automation server, with each input modality (click, type, keys, scroll, drag) and perception path (observe, wait_for, locate) earning its place. No redundant or filler tools.

    Completeness4/5

    The surface covers perception, waiting, all major input primitives, app lifecycle, and a grounding fallback, which is strong for GUI control. Minor gaps remain, such as explicit double-click/hover, clipboard, or screenshot-only capture, though observe's som/screenshot modes partially mitigate this.

    Maintenance

    ActivitySlowing
    ResponsivenessNo recent community issues

    Resources

    Related MCP Connectors

    Related MCP Servers

    • open-computer-use

      valkia
      A
      license
      Not graded
      quality
      C
      maintenance
      Enables the model to observe and control the live desktop via accessibility trees and screenshots, performing actions like clicking, typing, scrolling, dragging, and setting values.
      MIT

    • Open Computer Use

      aibo204
      A
      license
      Not graded
      quality
      B
      maintenance
      Enables AI agents to observe and control the user's live desktop by listing apps, reading accessibility trees, and performing clicks, typing, scrolling, dragging, and other input actions with per-action approvals and local audit archives.
      1
      MIT

    • dsh-computer-use

      bluechonk
      A
      license
      C
      quality
      C
      maintenance
      Enables agent-native desktop control by driving mouse, keyboard, and interface elements through accessibility-first semantic operations, with fallback to screen coordinates when accessibility cannot reach a target.
      30
      MIT

    • Windows Computer Bridge MCP

      Longtran2404
      A
      license
      Not graded
      quality
      B
      maintenance
      Enables AI clients to inspect and control allowlisted Windows applications using screenshots, mouse actions, Unicode keyboard input, UI Automation, and an Esc/F12 emergency stop.
      1
      Apache 2.0