1. Click on "Deploy Server". 2. Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state. 3. In the chat, type @ followed by the MCP server name and your instructions, e.g., "@Gorked Open Chrome, search cats" That's it! The server will respond to your query, and you can continue using it as needed. Here is a step-by-step guide with screenshots.

Gorked

by iBerry420

Overview Schema Source Related Servers Discussions Admin

Python

Local

Gorked

Computer-use agent framework for Grok Build CLI: see the screen, box every text region, target center pixels, and act — with a transparent thought process and a high-tech chat HUD.

Gorked does not call the xAI HTTP API for planning. The brain is the local grok binary (headless mode: streaming thoughts, tools, MCP). The hands and eyes are Gorked (OCR, mouse/keyboard, safety gates).

┌──────────────┐    ┌─────────────┐    ┌──────────────────┐    ┌─────────────────┐
│  Capture     │ →  │  OCR text   │ →  │  Headless Grok   │ →  │  Mouse/keyboard │
│  primary mon │    │  boxes+ctr  │    │  (CLI + MCP)     │    │  (dry-run/live) │
└──────────────┘    └─────────────┘    └──────────────────┘    └─────────────────┘
                                              │
                                              ▼
                               ┌──────────────────────────────┐
                               │  gorked-hud (C++ chat panel) │
                               │  thoughts · plans · actions  │
                               └──────────────────────────────┘

Features

Area	What you get
Screen vision	Fast screenshots (primary monitor only), RapidOCR boxes, center-pixel click targets
Annotation	Color boxes, labels, crosshairs + `(x,y)` chips on PNGs
Computer control	Move, click, double/right-click, type, click-and-type, hotkey, drag, scroll
Safety	Dry-run by default; live needs explicit flags + confirmation phrase
Brain	Headless `grok` CLI — streaming thoughts, tool use, JSON-schema plans
MCP tools	Grok drives Gorked via MCP (`analyze_screen`, `click_text`, `hotkey`, …)
HUD	Frameless C++ messenger (bottom-right): chat in, thoughts out, Esc aborts
Thought journal	OBSERVE / REASON / PLAN / ACT / RESULT traces → JSON + HUD bubbles

Related MCP server: ControlMCP

Requirements

Linux with X11 (Wayland-only sessions may need XWayland for capture/input)
Python 3.11+
Grok Build CLI on PATH (grok), authenticated (grok login)
C++17 toolchain + CMake (only if you build the HUD)
X11 libraries: libX11, libXft, libXrender, libXrandr, fontconfig, freetype

Optional: dual-monitor setups are fine — only the primary display is captured (secondary/TV ignored to save tokens).

Install

git clone https://github.com/iBerry420/gorked.git
cd gorked

python3 -m venv .venv
source .venv/bin/activate
pip install -e .

# Register project MCP for Grok Build
gorked mcp --install
# Trust the folder once if needed: open with `grok --trust` in this repo

Build the HUD (recommended)

cmake -S hud -B hud/build
cmake --build hud/build -j
# binary: hud/build/gorked-hud

Quick start

# Synthetic UI + OCR + heuristic plan (no Grok call required)
gorked demo

# Live OCR: boxes + center coordinates on primary monitor
gorked analyze
gorked monitors          # which display is captured

# Find UI text → absolute click coords
gorked find "Save"

# High-tech chat HUD (leave open)
gorked hud
# Live mouse/keyboard when you send tasks from the HUD:
gorked hud --live

# Headless Grok + MCP tools (dry-run)
gorked drive "Click the Settings button"

# Live computer use (real input)
gorked drive "Open Chrome, search cats" --live

HUD controls

Input	Action
Enter	Newline in composer
Ctrl+Enter	Send task → spawns `gorked drive`
Esc	Abort running agent process tree
Drag header	Move window (frameless tool window)
−	Minimize to pill (flush bottom-right)
Restore (on pill)	Expand full panel
×	Quit

Architecture

Two ways to run the agent

1. Drive mode (recommended) — Grok is in charge via MCP:

User / HUD  →  gorked drive "goal"  →  grok -p (streaming-json, --yolo, --trust)
                                         ↓ MCP tool calls
                                    gorked MCP server
                                         ↓
                              capture · OCR · mouse · keyboard
                                         ↓
                              output/ + HUD bubbles

2. Agent/plan mode — Gorked OCRs, Grok returns structured PlannedAction[], Gorked executes:

gorked agent "goal"   # or gorked plan "goal"

Primary-only capture

mss index	Meaning	Default
0	Virtual all-monitors	Blocked
1…	Physical displays	Primary only (`is_primary`)

Env overrides: GORKED_ALLOW_ALL_MONITORS=1, GORKED_ALLOW_MONITOR_OVERRIDE=1, GORKED_MONITOR=primary.

Live vs dry-run

Layer	Dry-run	Live
CLI	default	`--live`
HUD	`gorked hud`	`gorked hud --live`
MCP process env	`GORKED_DRY_RUN=1`	`GORKED_LIVE=1` + `GORKED_DRY_RUN=0`
Explicit toggle	—	`set_live_mode(true, confirm_phrase="I_ACCEPT_LIVE_CONTROL")`

Live mode is not pinned in .grok/config.toml so each run inherits the correct mode from the parent process.

MCP tools

Exposed to headless Grok as server gorked:

Tool	Purpose
`get_status`	dry_run / live, monitor, paths
`set_live_mode`	Gate real input (or no-op if already live)
`thought`	User-visible reasoning → HUD
`list_monitors` / `capture_screen`	Display info / raw shot
`analyze_screen` / `analyze_image`	OCR boxes + centers
`find_text` / `click_text`	Locate / click labels
`mouse_move` / `mouse_click` / `mouse_drag` / `mouse_scroll`	Pointer
`type_text` / `click_and_type` / `hotkey`	Keyboard
`wait_ms` / `recent_actions`	Timing / audit

grok mcp list
grok mcp doctor gorked

CLI reference

gorked capture [options]      Screenshot primary monitor
gorked analyze [image]        OCR + annotate + scene JSON
gorked find QUERY             Match text → center coords
gorked plan GOAL              Plan only (Grok CLI / heuristic)
gorked agent GOAL             Observe → plan → act loop
gorked drive GOAL             Full Grok+MCP computer-use loop
gorked demo                   Synthetic UI end-to-end
gorked monitors               List displays + capture policy
gorked mcp [--install]        Run or register MCP server
gorked hud [--live]           Launch C++ chat HUD

Useful flags: --out, --monitor, --heuristic, --no-tools, --model, --max-turns, --json.

Python API

from gorked.pipeline import VisionPipeline
from gorked.agent import AgentConfig, ComputerUseAgent
from gorked.drive import run_drive
from gorked import computer

# Vision
pipe = VisionPipeline("output")
scene, raw, annotated, journal = pipe.analyze_screen()
for r in scene.regions:
    print(r.text, r.click_point.as_tuple())

# Structured agent (dry-run)
agent = ComputerUseAgent(AgentConfig(goal="Click Settings", dry_run=True))
result = agent.run()

# Drive with Grok + MCP
run_drive("Open the browser settings", live=False)

# Low-level actions
ctrl = computer.ComputerController(dry_run=True)
ctrl.execute(computer.click_at(100, 200, rationale="example"))

Project layout

gorked/
├── src/gorked/
│   ├── capture.py      # mss screenshots (primary-only policy)
│   ├── ocr.py          # RapidOCR → TextRegion + center
│   ├── annotate.py     # boxes / crosshairs
│   ├── computer.py     # pynput mouse + keyboard
│   ├── models.py       # BBox, ScreenScene, PlannedAction, …
│   ├── thought.py      # thought journal → HUD bus
│   ├── hud_bus.py      # JSONL events for the C++ HUD
│   ├── grok_cli.py     # headless `grok` runner (streaming-json)
│   ├── planner.py      # structured plans + heuristic fallback
│   ├── agent.py        # observe → plan → act loop
│   ├── drive.py        # Grok + MCP drive mode
│   ├── mcp_server.py   # FastMCP tool surface
│   ├── pipeline.py     # capture → detect → annotate
│   └── cli.py          # typer entrypoints
├── hud/                # C++17 X11/Xft frameless messenger
│   ├── src/
│   ├── CMakeLists.txt
│   └── README.md
├── .grok/skills/       # computer-use skill for Grok Build
├── AGENTS.md           # project rules for Grok
├── pyproject.toml
└── README.md

Artifacts under output/ (gitignored): captures, annotated PNGs, scene JSON, thought traces, hud.jsonl.

Environment variables

Variable	Purpose
`GORKED_MONITOR`	`primary` (default) or index
`GORKED_LIVE`	`1` → real input at MCP boot
`GORKED_DRY_RUN`	`1` → simulate input
`GORKED_OUTPUT_DIR`	Artifact directory
`GORKED_HUD_STREAM`	Path to HUD JSONL stream
`GORKED_HUD_LIVE`	HUD-spawned drives use `--live`
`GORKED_BIN`	Path to `gorked` CLI for the HUD
`GORKED_GROK_BIN`	Path to `grok` if not on PATH
`GORKED_ALLOW_ALL_MONITORS`	Allow virtual dual-desktop capture
`GORKED_ALLOW_MONITOR_OVERRIDE`	Allow non-primary indices

Copy .env.example for a starting point (never commit real secrets).

Safety notes

Dry-run is the default. Live control can move the mouse and type for real.
Prefer keeping the Grok TUI / HUD from covering the UI the agent must click.
Esc in the HUD aborts the active drive process group (SIGTERM → SIGKILL).
Do not run live mode unattended on sensitive sessions without allowlists.

Development

source .venv/bin/activate
pip install -e ".[dev]"

# Reinstall after code changes (editable)
pip install -e .

# Rebuild HUD
cmake --build hud/build -j

Roadmap

Capture + OCR boxes + center click targets
Computer primitives + dry-run
Thought journal + HUD stream
Headless Grok CLI planner + drive mode
MCP tool surface for Grok
Frameless C++ chat HUD (send / abort / live)
Multi-step re-OCR verification loops
Action allowlists / forbidden zones
Native Grok computer-use handoff when available

License

MIT — see LICENSE.

Acknowledgments

Grok Build / xAI — headless agent CLI
RapidOCR — ONNX text detection
mss — screenshots
pynput — input
MCP — tool protocol

This server cannot be deployed

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Maintenance

ActivitySlowing

ResponsivenessNo issues

Resources

GitHub Repository

Need Help?

Related Servers

Related MCP Connectors

Frenchie A
io.github.Lab94
OCR, transcription, file extraction, and image generation for AI agents via MCP.
AI & Machine Learning Image & Video Processing Speech Processing
72
Great ArrowOAuth
ai.greatarrow
Shared memory and actions for Claude, Kiro, OpenAI, Cursor, and other MCP-compatible AI clients.
Knowledge & Memory
1
llm-orchestration-agent-mcp
ai.getvda
LLM Orchestration Agent (Mcp)
Agent Orchestration Autonomous Agents
1
DazbenchOAuth
app.dazbench
Task management your AI agents can actually run. One line becomes a context-ready task over MCP.
Autonomous Agents Project Management Agent Orchestration
1

Related MCP Servers

openowl
mihir-kanzariya
A
license
Not graded
quality
B
maintenance
An MCP server that gives any AI assistant eyes and hands on your desktop — screenshots, clicking, typing, OCR, window management, accessibility-tree queries, workflow recording.
Updated 4 months ago (2026-04-30 07:38 UTC)
5
Apache 2.0
ControlMCP
OS Automation Browser Automation
nix18
A
license
Not graded
quality
D
maintenance
An MCP server that enables LLMs to see and control a computer — screen capture, window management, mouse and keyboard automation — with a structured plan-execute workflow for complex desktop automation.
Updated 6 months ago (2026-03-27 08:33 UTC)
GPL 3.0
computer-use-mcp
OS Automation Autonomous Agents
yangweijie
A
license
Not graded
quality
C
maintenance
A framework-agnostic computer-use MCP server that exposes core desktop operations (screen capture, mouse, keyboard, and file access) as standard MCP tools, enabling any MCP-compatible agent to drive a computer.
Updated 2 months ago (2026-07-28 09:13 UTC)
236 npm
MIT
da-mcp
OS Automation Image & Video Processing Command Line
cioinside
F
license
A
quality
A
maintenance
Cross-platform desktop automation MCP server that lets AI agents capture screenshots, run OCR with UI-element classification, control mouse/keyboard, and launch programs on Linux, macOS, and Windows.
Updated a month ago (2026-08-08 19:47 UTC)
20
1
-

Gorked

Features

Requirements

Install

Build the HUD (recommended)

Quick start

HUD controls

Architecture

Two ways to run the agent

Primary-only capture

Live vs dry-run

MCP tools

CLI reference

Python API

Project layout

Environment variables

Safety notes

Development

Roadmap

License

Acknowledgments

Maintenance

Resources

Related MCP Connectors

Related MCP Servers

openowl

ControlMCP

computer-use-mcp

da-mcp