1. Click on "Install Server". 2. Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state. 3. In the chat, type @ followed by the MCP server name and your instructions, e.g., "@Gorked Open Chrome, search cats" That's it! The server will respond to your query, and you can continue using it as needed. Here is a step-by-step guide with screenshots.

Gorked

by iBerry420

Overview Schema Related Servers Score Discussions

Python

Local

Gorked

Computer-use agent framework for Grok Build CLI: see the screen, box every text region, target center pixels, and act — with a transparent thought process and a high-tech chat HUD.

Gorked does not call the xAI HTTP API for planning. The brain is the local grok binary (headless mode: streaming thoughts, tools, MCP). The hands and eyes are Gorked (OCR, mouse/keyboard, safety gates).

┌──────────────┐    ┌─────────────┐    ┌──────────────────┐    ┌─────────────────┐
│  Capture     │ →  │  OCR text   │ →  │  Headless Grok   │ →  │  Mouse/keyboard │
│  primary mon │    │  boxes+ctr  │    │  (CLI + MCP)     │    │  (dry-run/live) │
└──────────────┘    └─────────────┘    └──────────────────┘    └─────────────────┘
                                              │
                                              ▼
                               ┌──────────────────────────────┐
                               │  gorked-hud (C++ chat panel) │
                               │  thoughts · plans · actions  │
                               └──────────────────────────────┘

Features

Area	What you get
Screen vision	Fast screenshots (primary monitor only), RapidOCR boxes, center-pixel click targets
Annotation	Color boxes, labels, crosshairs + `(x,y)` chips on PNGs
Computer control	Move, click, double/right-click, type, click-and-type, hotkey, drag, scroll
Safety	Dry-run by default; live needs explicit flags + confirmation phrase
Brain	Headless `grok` CLI — streaming thoughts, tool use, JSON-schema plans
MCP tools	Grok drives Gorked via MCP (`analyze_screen`, `click_text`, `hotkey`, …)
HUD	Frameless C++ messenger (bottom-right): chat in, thoughts out, Esc aborts
Thought journal	OBSERVE / REASON / PLAN / ACT / RESULT traces → JSON + HUD bubbles

Related MCP server: helix-pilot

Requirements

Linux with X11 (Wayland-only sessions may need XWayland for capture/input)
Python 3.11+
Grok Build CLI on PATH (grok), authenticated (grok login)
C++17 toolchain + CMake (only if you build the HUD)
X11 libraries: libX11, libXft, libXrender, libXrandr, fontconfig, freetype

Optional: dual-monitor setups are fine — only the primary display is captured (secondary/TV ignored to save tokens).

Install

git clone https://github.com/iBerry420/gorked.git
cd gorked

python3 -m venv .venv
source .venv/bin/activate
pip install -e .

# Register project MCP for Grok Build
gorked mcp --install
# Trust the folder once if needed: open with `grok --trust` in this repo

Build the HUD (recommended)

cmake -S hud -B hud/build
cmake --build hud/build -j
# binary: hud/build/gorked-hud

Quick start

# Synthetic UI + OCR + heuristic plan (no Grok call required)
gorked demo

# Live OCR: boxes + center coordinates on primary monitor
gorked analyze
gorked monitors          # which display is captured

# Find UI text → absolute click coords
gorked find "Save"

# High-tech chat HUD (leave open)
gorked hud
# Live mouse/keyboard when you send tasks from the HUD:
gorked hud --live

# Headless Grok + MCP tools (dry-run)
gorked drive "Click the Settings button"

# Live computer use (real input)
gorked drive "Open Chrome, search cats" --live

HUD controls

Input	Action
Enter	Newline in composer
Ctrl+Enter	Send task → spawns `gorked drive`
Esc	Abort running agent process tree
Drag header	Move window (frameless tool window)
−	Minimize to pill (flush bottom-right)
Restore (on pill)	Expand full panel
×	Quit

Architecture

Two ways to run the agent

1. Drive mode (recommended) — Grok is in charge via MCP:

User / HUD  →  gorked drive "goal"  →  grok -p (streaming-json, --yolo, --trust)
                                         ↓ MCP tool calls
                                    gorked MCP server
                                         ↓
                              capture · OCR · mouse · keyboard
                                         ↓
                              output/ + HUD bubbles

2. Agent/plan mode — Gorked OCRs, Grok returns structured PlannedAction[], Gorked executes:

gorked agent "goal"   # or gorked plan "goal"

Primary-only capture

mss index	Meaning	Default
0	Virtual all-monitors	Blocked
1…	Physical displays	Primary only (`is_primary`)

Env overrides: GORKED_ALLOW_ALL_MONITORS=1, GORKED_ALLOW_MONITOR_OVERRIDE=1, GORKED_MONITOR=primary.

Live vs dry-run

Layer	Dry-run	Live
CLI	default	`--live`
HUD	`gorked hud`	`gorked hud --live`
MCP process env	`GORKED_DRY_RUN=1`	`GORKED_LIVE=1` + `GORKED_DRY_RUN=0`
Explicit toggle	—	`set_live_mode(true, confirm_phrase="I_ACCEPT_LIVE_CONTROL")`

Live mode is not pinned in .grok/config.toml so each run inherits the correct mode from the parent process.

MCP tools

Exposed to headless Grok as server gorked:

Tool	Purpose
`get_status`	dry_run / live, monitor, paths
`set_live_mode`	Gate real input (or no-op if already live)
`thought`	User-visible reasoning → HUD
`list_monitors` / `capture_screen`	Display info / raw shot
`analyze_screen` / `analyze_image`	OCR boxes + centers
`find_text` / `click_text`	Locate / click labels
`mouse_move` / `mouse_click` / `mouse_drag` / `mouse_scroll`	Pointer
`type_text` / `click_and_type` / `hotkey`	Keyboard
`wait_ms` / `recent_actions`	Timing / audit

grok mcp list
grok mcp doctor gorked

CLI reference

gorked capture [options]      Screenshot primary monitor
gorked analyze [image]        OCR + annotate + scene JSON
gorked find QUERY             Match text → center coords
gorked plan GOAL              Plan only (Grok CLI / heuristic)
gorked agent GOAL             Observe → plan → act loop
gorked drive GOAL             Full Grok+MCP computer-use loop
gorked demo                   Synthetic UI end-to-end
gorked monitors               List displays + capture policy
gorked mcp [--install]        Run or register MCP server
gorked hud [--live]           Launch C++ chat HUD

Useful flags: --out, --monitor, --heuristic, --no-tools, --model, --max-turns, --json.

Python API

from gorked.pipeline import VisionPipeline
from gorked.agent import AgentConfig, ComputerUseAgent
from gorked.drive import run_drive
from gorked import computer

# Vision
pipe = VisionPipeline("output")
scene, raw, annotated, journal = pipe.analyze_screen()
for r in scene.regions:
    print(r.text, r.click_point.as_tuple())

# Structured agent (dry-run)
agent = ComputerUseAgent(AgentConfig(goal="Click Settings", dry_run=True))
result = agent.run()

# Drive with Grok + MCP
run_drive("Open the browser settings", live=False)

# Low-level actions
ctrl = computer.ComputerController(dry_run=True)
ctrl.execute(computer.click_at(100, 200, rationale="example"))

Project layout

gorked/
├── src/gorked/
│   ├── capture.py      # mss screenshots (primary-only policy)
│   ├── ocr.py          # RapidOCR → TextRegion + center
│   ├── annotate.py     # boxes / crosshairs
│   ├── computer.py     # pynput mouse + keyboard
│   ├── models.py       # BBox, ScreenScene, PlannedAction, …
│   ├── thought.py      # thought journal → HUD bus
│   ├── hud_bus.py      # JSONL events for the C++ HUD
│   ├── grok_cli.py     # headless `grok` runner (streaming-json)
│   ├── planner.py      # structured plans + heuristic fallback
│   ├── agent.py        # observe → plan → act loop
│   ├── drive.py        # Grok + MCP drive mode
│   ├── mcp_server.py   # FastMCP tool surface
│   ├── pipeline.py     # capture → detect → annotate
│   └── cli.py          # typer entrypoints
├── hud/                # C++17 X11/Xft frameless messenger
│   ├── src/
│   ├── CMakeLists.txt
│   └── README.md
├── .grok/skills/       # computer-use skill for Grok Build
├── AGENTS.md           # project rules for Grok
├── pyproject.toml
└── README.md

Artifacts under output/ (gitignored): captures, annotated PNGs, scene JSON, thought traces, hud.jsonl.

Environment variables

Variable	Purpose
`GORKED_MONITOR`	`primary` (default) or index
`GORKED_LIVE`	`1` → real input at MCP boot
`GORKED_DRY_RUN`	`1` → simulate input
`GORKED_OUTPUT_DIR`	Artifact directory
`GORKED_HUD_STREAM`	Path to HUD JSONL stream
`GORKED_HUD_LIVE`	HUD-spawned drives use `--live`
`GORKED_BIN`	Path to `gorked` CLI for the HUD
`GORKED_GROK_BIN`	Path to `grok` if not on PATH
`GORKED_ALLOW_ALL_MONITORS`	Allow virtual dual-desktop capture
`GORKED_ALLOW_MONITOR_OVERRIDE`	Allow non-primary indices

Copy .env.example for a starting point (never commit real secrets).

Safety notes

Dry-run is the default. Live control can move the mouse and type for real.
Prefer keeping the Grok TUI / HUD from covering the UI the agent must click.
Esc in the HUD aborts the active drive process group (SIGTERM → SIGKILL).
Do not run live mode unattended on sensitive sessions without allowlists.

Development

source .venv/bin/activate
pip install -e ".[dev]"

# Reinstall after code changes (editable)
pip install -e .

# Rebuild HUD
cmake --build hud/build -j

Roadmap

Capture + OCR boxes + center click targets
Computer primitives + dry-run
Thought journal + HUD stream
Headless Grok CLI planner + drive mode
MCP tool surface for Grok
Frameless C++ chat HUD (send / abort / live)
Multi-step re-OCR verification loops
Action allowlists / forbidden zones
Native Grok computer-use handoff when available

License

MIT — see LICENSE.

Acknowledgments

Grok Build / xAI — headless agent CLI
RapidOCR — ONNX text detection
mss — screenshots
pynput — input
MCP — tool protocol

This server cannot be installed

license - permissive license

quality - not tested

maintenance - not tested

How are these scores calculated?

Resources

GitHub Repository

Need Help?

Related Servers

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Latest Blog Posts

Your AI Chatbot Just Exposed Your CEO's Salary to an Intern
By Om-Shree-0709 on July 2, 2026.
Agent Identity
MCP Security
OAuth Delegation
Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)
By Om-Shree-0709 on June 30, 2026.
Agentic Ai
Prompt Injection
WebAssembly
Lightport: Open-Sourcing Glama's AI Gateway
By punkpeye on April 27, 2026.
OpenAI
open source

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/iBerry420/gorked'

If you have feedback or need assistance with the MCP directory API, please join our Discord server