# Gorked

**Computer-use agent framework** for [Grok Build](https://x.ai) CLI: see the screen, box every text region, target center pixels, and act — with a transparent thought process and a high-tech chat HUD.

Gorked does **not** call the xAI HTTP API for planning. The brain is the local **`grok` binary** (headless mode: streaming thoughts, tools, MCP). The hands and eyes are Gorked (OCR, mouse/keyboard, safety gates).

```
┌──────────────┐    ┌─────────────┐    ┌──────────────────┐    ┌─────────────────┐
│  Capture     │ →  │  OCR text   │ →  │  Headless Grok   │ →  │  Mouse/keyboard │
│  primary mon │    │  boxes+ctr  │    │  (CLI + MCP)     │    │  (dry-run/live) │
└──────────────┘    └─────────────┘    └──────────────────┘    └─────────────────┘
                                              │
                                              ▼
                               ┌──────────────────────────────┐
                               │  gorked-hud (C++ chat panel) │
                               │  thoughts · plans · actions  │
                               └──────────────────────────────┘
```

---

## Features

| Area | What you get |
|------|----------------|
| **Screen vision** | Fast screenshots (primary monitor only), RapidOCR boxes, center-pixel click targets |
| **Annotation** | Color boxes, labels, crosshairs + `(x,y)` chips on PNGs |
| **Computer control** | Move, click, double/right-click, type, click-and-type, hotkey, drag, scroll |
| **Safety** | Dry-run by default; live needs explicit flags + confirmation phrase |
| **Brain** | Headless `grok` CLI — streaming thoughts, tool use, JSON-schema plans |
| **MCP tools** | Grok drives Gorked via MCP (`analyze_screen`, `click_text`, `hotkey`, …) |
| **HUD** | Frameless C++ messenger (bottom-right): chat in, thoughts out, Esc aborts |
| **Thought journal** | OBSERVE / REASON / PLAN / ACT / RESULT traces → JSON + HUD bubbles |

---

## Requirements

- **Linux** with **X11** (Wayland-only sessions may need XWayland for capture/input)
- **Python 3.11+**
- **[Grok Build CLI](https://x.ai)** on `PATH` (`grok`), authenticated (`grok login`)
- **C++17 toolchain** + CMake (only if you build the HUD)
- X11 libraries: `libX11`, `libXft`, `libXrender`, `libXrandr`, fontconfig, freetype

Optional: dual-monitor setups are fine — **only the primary display is captured** (secondary/TV ignored to save tokens).

---

## Install

```bash
git clone https://github.com/iBerry420/gorked.git
cd gorked

python3 -m venv .venv
source .venv/bin/activate
pip install -e .

# Register project MCP for Grok Build
gorked mcp --install
# Trust the folder once if needed: open with `grok --trust` in this repo
```

### Build the HUD (recommended)

```bash
cmake -S hud -B hud/build
cmake --build hud/build -j
# binary: hud/build/gorked-hud
```

---

## Quick start

```bash
# Synthetic UI + OCR + heuristic plan (no Grok call required)
gorked demo

# Live OCR: boxes + center coordinates on primary monitor
gorked analyze
gorked monitors          # which display is captured

# Find UI text → absolute click coords
gorked find "Save"

# High-tech chat HUD (leave open)
gorked hud
# Live mouse/keyboard when you send tasks from the HUD:
gorked hud --live

# Headless Grok + MCP tools (dry-run)
gorked drive "Click the Settings button"

# Live computer use (real input)
gorked drive "Open Chrome, search cats" --live
```

### HUD controls

| Input | Action |
|-------|--------|
| **Enter** | Newline in composer |
| **Ctrl+Enter** | Send task → spawns `gorked drive` |
| **Esc** | Abort running agent process tree |
| Drag header | Move window (frameless tool window) |
| **−** | Minimize to pill (flush bottom-right) |
| **Restore** (on pill) | Expand full panel |
| **×** | Quit |

---

## Architecture

### Two ways to run the agent

**1. Drive mode (recommended)** — Grok is in charge via MCP:

```
User / HUD  →  gorked drive "goal"  →  grok -p (streaming-json, --yolo, --trust)
                                         ↓ MCP tool calls
                                    gorked MCP server
                                         ↓
                              capture · OCR · mouse · keyboard
                                         ↓
                              output/ + HUD bubbles
```

**2. Agent/plan mode** — Gorked OCRs, Grok returns structured `PlannedAction[]`, Gorked executes:

```
gorked agent "goal"   # or gorked plan "goal"
```

### Primary-only capture

| mss index | Meaning | Default |
|-----------|---------|---------|
| 0 | Virtual all-monitors | **Blocked** |
| 1… | Physical displays | Primary only (`is_primary`) |

Env overrides: `GORKED_ALLOW_ALL_MONITORS=1`, `GORKED_ALLOW_MONITOR_OVERRIDE=1`, `GORKED_MONITOR=primary`.

### Live vs dry-run

| Layer | Dry-run | Live |
|-------|---------|------|
| CLI | default | `--live` |
| HUD | `gorked hud` | `gorked hud --live` |
| MCP process env | `GORKED_DRY_RUN=1` | `GORKED_LIVE=1` + `GORKED_DRY_RUN=0` |
| Explicit toggle | — | `set_live_mode(true, confirm_phrase="I_ACCEPT_LIVE_CONTROL")` |

Live mode is **not** pinned in `.grok/config.toml` so each run inherits the correct mode from the parent process.

---

## MCP tools

Exposed to headless Grok as server `gorked`:

| Tool | Purpose |
|------|---------|
| `get_status` | dry_run / live, monitor, paths |
| `set_live_mode` | Gate real input (or no-op if already live) |
| `thought` | User-visible reasoning → HUD |
| `list_monitors` / `capture_screen` | Display info / raw shot |
| `analyze_screen` / `analyze_image` | OCR boxes + centers |
| `find_text` / `click_text` | Locate / click labels |
| `mouse_move` / `mouse_click` / `mouse_drag` / `mouse_scroll` | Pointer |
| `type_text` / `click_and_type` / `hotkey` | Keyboard |
| `wait_ms` / `recent_actions` | Timing / audit |

```bash
grok mcp list
grok mcp doctor gorked
```

---

## CLI reference

```
gorked capture [options]      Screenshot primary monitor
gorked analyze [image]        OCR + annotate + scene JSON
gorked find QUERY             Match text → center coords
gorked plan GOAL              Plan only (Grok CLI / heuristic)
gorked agent GOAL             Observe → plan → act loop
gorked drive GOAL             Full Grok+MCP computer-use loop
gorked demo                   Synthetic UI end-to-end
gorked monitors               List displays + capture policy
gorked mcp [--install]        Run or register MCP server
gorked hud [--live]           Launch C++ chat HUD
```

Useful flags: `--out`, `--monitor`, `--heuristic`, `--no-tools`, `--model`, `--max-turns`, `--json`.

---

## Python API

```python
from gorked.pipeline import VisionPipeline
from gorked.agent import AgentConfig, ComputerUseAgent
from gorked.drive import run_drive
from gorked import computer

# Vision
pipe = VisionPipeline("output")
scene, raw, annotated, journal = pipe.analyze_screen()
for r in scene.regions:
    print(r.text, r.click_point.as_tuple())

# Structured agent (dry-run)
agent = ComputerUseAgent(AgentConfig(goal="Click Settings", dry_run=True))
result = agent.run()

# Drive with Grok + MCP
run_drive("Open the browser settings", live=False)

# Low-level actions
ctrl = computer.ComputerController(dry_run=True)
ctrl.execute(computer.click_at(100, 200, rationale="example"))
```

---

## Project layout

```
gorked/
├── src/gorked/
│   ├── capture.py      # mss screenshots (primary-only policy)
│   ├── ocr.py          # RapidOCR → TextRegion + center
│   ├── annotate.py     # boxes / crosshairs
│   ├── computer.py     # pynput mouse + keyboard
│   ├── models.py       # BBox, ScreenScene, PlannedAction, …
│   ├── thought.py      # thought journal → HUD bus
│   ├── hud_bus.py      # JSONL events for the C++ HUD
│   ├── grok_cli.py     # headless `grok` runner (streaming-json)
│   ├── planner.py      # structured plans + heuristic fallback
│   ├── agent.py        # observe → plan → act loop
│   ├── drive.py        # Grok + MCP drive mode
│   ├── mcp_server.py   # FastMCP tool surface
│   ├── pipeline.py     # capture → detect → annotate
│   └── cli.py          # typer entrypoints
├── hud/                # C++17 X11/Xft frameless messenger
│   ├── src/
│   ├── CMakeLists.txt
│   └── README.md
├── .grok/skills/       # computer-use skill for Grok Build
├── AGENTS.md           # project rules for Grok
├── pyproject.toml
└── README.md
```

Artifacts under `output/` (gitignored): captures, annotated PNGs, scene JSON, thought traces, `hud.jsonl`.

---

## Environment variables

| Variable | Purpose |
|----------|---------|
| `GORKED_MONITOR` | `primary` (default) or index |
| `GORKED_LIVE` | `1` → real input at MCP boot |
| `GORKED_DRY_RUN` | `1` → simulate input |
| `GORKED_OUTPUT_DIR` | Artifact directory |
| `GORKED_HUD_STREAM` | Path to HUD JSONL stream |
| `GORKED_HUD_LIVE` | HUD-spawned drives use `--live` |
| `GORKED_BIN` | Path to `gorked` CLI for the HUD |
| `GORKED_GROK_BIN` | Path to `grok` if not on PATH |
| `GORKED_ALLOW_ALL_MONITORS` | Allow virtual dual-desktop capture |
| `GORKED_ALLOW_MONITOR_OVERRIDE` | Allow non-primary indices |

Copy `.env.example` for a starting point (never commit real secrets).

---

## Safety notes

- **Dry-run is the default.** Live control can move the mouse and type for real.
- Prefer keeping the Grok TUI / HUD from covering the UI the agent must click.
- Esc in the HUD aborts the active drive process group (`SIGTERM` → `SIGKILL`).
- Do not run live mode unattended on sensitive sessions without allowlists.

---

## Development

```bash
source .venv/bin/activate
pip install -e ".[dev]"

# Reinstall after code changes (editable)
pip install -e .

# Rebuild HUD
cmake --build hud/build -j
```

---

## Roadmap

- [x] Capture + OCR boxes + center click targets  
- [x] Computer primitives + dry-run  
- [x] Thought journal + HUD stream  
- [x] Headless Grok CLI planner + drive mode  
- [x] MCP tool surface for Grok  
- [x] Frameless C++ chat HUD (send / abort / live)  
- [ ] Multi-step re-OCR verification loops  
- [ ] Action allowlists / forbidden zones  
- [ ] Native Grok computer-use handoff when available  

---

## License

MIT — see [LICENSE](LICENSE).

---

## Acknowledgments

- [Grok Build](https://x.ai) / xAI — headless agent CLI  
- [RapidOCR](https://github.com/RapidAI/RapidOCR) — ONNX text detection  
- [mss](https://github.com/BoboTiG/python-mss) — screenshots  
- [pynput](https://github.com/moses-palmer/pynput) — input  
- [MCP](https://modelcontextprotocol.io) — tool protocol