# astra-linux-agent

**Linux computer-use capability layer for GPT-6 Astra running through Codex with ChatGPT subscription authentication — Wayland/Hyprland first.**

astra-linux-agent is a local [MCP](https://modelcontextprotocol.io) server that gives Astra *eyes and hands* on a Linux desktop:
screenshots and structured desktop state, pointer and keyboard input, semantic window/workspace control, a
Chromium browser lane, and structured Linux tools (filesystem, git, processes, systemd, containers, clipboard) —
all behind one cross-tool **policy engine** that judges actions by consequence, whichever route the model takes.

> **Not an official OpenAI product.** This is an independent open-source project. It is not affiliated with,
> endorsed by, or supported by OpenAI. "ChatGPT", "Codex" and "Astra" are OpenAI's names for their products.

## How it connects (no API key)

```
GPT-6 Astra
    │  your ChatGPT subscription
Codex CLI  — "Sign in with ChatGPT"
    │  MCP over stdio (Codex launches `astra-agent mcp`)
┌───▼─────────────────────────────────────────┐
│  astra-linux-agent                          │
│   policy engine  → capability router        │
│   ┌────────────┬────────────┬─────────────┐ │
│   │ structured │  desktop   │  computer   │ │
│   │ shell/git  │ Hyprland   │ screenshot  │ │
│   │ fs/systemd │ windows    │ click/type  │ │
│   │ containers │ workspaces │ scroll/drag │ │
│   │ browser    │ events     │ visual QA   │ │
│   └────────────┴────────────┴─────────────┘ │
└──────────────────┬──────────────────────────┘
              Wayland / Hyprland
```

* An **eligible ChatGPT plan** is required to use Astra through Codex. Codex authenticates with *Sign in with ChatGPT*.
* **No OpenAI API credentials are required or used.** This project never calls the OpenAI API and has no place to put an API key.
* It does not proxy, spoof, scrape, or circumvent ChatGPT authentication; it uses only supported Codex mechanisms
  (ChatGPT login, MCP servers, MCP elicitation, Codex approval modes).

Verified with codex-cli 0.153.4 and `gpt-6-astra`: screenshots returned as MCP image content are seen by the model
(image-only, text+image and image+structuredContent), a screenshot → click → screenshot loop completes, and `view_image`
on a saved PNG works as a fallback. `gpt-6-astra` refuses codex-cli 0.151.0 ("requires a newer version of Codex").

## Why semantic tools before pixels

Clicking through a GUI is slow, brittle, and hard to audit. The router therefore exposes the **most semantic and
deterministic interface available** and only falls back to vision when the GUI is intrinsically required:

| Instead of… | Astra can call… |
|---|---|
| opening a terminal and typing `systemctl status foo` | `systemd_status` |
| hunting for a window in a task switcher | `desktop_focus_window` with a stable handle from `desktop_state` |
| clicking a launcher icon | `desktop_launch` |
| reading a web page from pixels | `browser_snapshot` / `browser_read_text` (loopback CDP) |
| scrolling and squinting at a diff | `git_diff` |

Vision (`computer_screenshot`, `computer_click`, `computer_type`, …) remains first-class for GUI-only dialogs,
graphical editors, canvases, drag-and-drop, and *verifying* that a semantic action actually produced the visible result.
Every `computer_*` action returns a fresh screenshot plus `DESKTOP_STATE` (windows with handles, workspaces,
monitors, cursor, lock state), so Astra always has **pixels + structured truth**.

## The policy engine sits above every lane

Policy classifies **intent by consequence**, not by tool name:

| Level | Examples | Default |
|---|---|---|
| 0 observe | screenshot, read file, list windows, `git status` | allow |
| 1 low | focus window, switch workspace, scroll, launch an app | allow |
| 2 local | write a file in a workspace root, local commit, change a preference | confirm |
| 3 consequential | `git push`, submit a form, send a message, stop a service | confirm |
| 4 privileged | `sudo`, disk tools, credentials, firewall/auth changes, `rm -rf` outside roots | deny (never grantable) |

The defining property: if policy denies `shell.run("rm -rf ~/important")`, it **also** denies Astra typing
`rm -rf ~/important` + Enter into a terminal window it opened visually. Typed text and key chords aimed at a
terminal-class window are inferred into the same `shell.run` intent and judged by the same rules; the input backend
never receives the keystrokes of a denied action.

**Honest boundary.** Terminal-intent inference is heuristic. It cannot see a command assembled from clipboard pastes,
shell-history recall, aliases, GUI "run command" dialogs, or a terminal emulator it does not recognise. Defence in
depth comes from the *access stages* below, Codex's own MCP approval mode, the audit log, and the emergency stop.
See [docs/policy.md](docs/policy.md).

### Access stages (start small)

| Stage | What Astra may act on |
|---|---|
| `observe` (default) | nothing — screenshots and structured state only |
| `headless` | only the agent's own virtual output (an isolated Hyprland headless display) |
| `workspace` | one named workspace |
| `desktop` | the whole desktop (policy rules still apply) |

## Quick start

Requirements: Linux, Wayland, **Hyprland ≥ 0.56** (tested 0.56.2), Python ≥ 3.11, `grim`, `wl-clipboard`,
and Codex CLI ≥ 0.153 signed in with ChatGPT. No root, no `/dev/uinput`, no daemon.

```bash
# 1. Codex, signed in with your ChatGPT account
codex login            # choose "Sign in with ChatGPT"

# 2. astra-linux-agent (from source until the first PyPI release)
uv tool install git+https://github.com/novique-ai/astra-linux-agent   # or: pipx install git+https://github.com/novique-ai/astra-linux-agent
astra-agent doctor                       # checks compositor, grim, Wayland protocols, Codex auth
astra-agent config init                  # writes ~/.config/astra-linux-agent/config.toml (observe-only)

# 3. Register with Codex
astra-agent codex install                # runs `codex mcp add …` and sets the approval mode

# 4. Use it
codex                                    # then: "Take a screenshot and tell me which windows are open."
```

Non-interactive runs: `codex exec --approve-for-me "…"` (Codex forces `approval_policy = never` in `exec`, so MCP
calls need its automatic reviewer).

Widen access deliberately, one step at a time:

```toml
# ~/.config/astra-linux-agent/config.toml
stage = "headless"                 # observe → headless → workspace → desktop
[display.headless]
enabled = true                     # isolated virtual output for the agent
```

See [docs/install.md](docs/install.md) and [docs/headless.md](docs/headless.md).

## Safety posture (read this)

This software gives an AI agent eyes and hands on your computer.

* Start in `observe`; move to `headless` before `desktop`.
* Never run it as root; it needs no privileges.
* Keep Codex's MCP approval mode at `prompt` or `writes` until you trust your policy; `auto` skips Codex-side prompts.
* Emergency stop: `astra-agent stop` (or `touch "$XDG_RUNTIME_DIR/astra-linux-agent/STOP"`); bind it to a key.
  **Human input wins**: if the cursor moves without the agent moving it, the agent pauses itself.
* The browser lane binds CDP to loopback on an ephemeral port with a throwaway profile; never expose it.
* The audit log (`~/.local/state/astra-linux-agent/audit.jsonl`) records every decision with secrets redacted;
  screenshots and clipboard contents are never logged.
* Screenshots leave your machine: they go to OpenAI through your Codex session, like any image you attach in Codex.

## What is in the box

| Subsystem | Implementation |
|---|---|
| Compositor | Hyprland IPC over its UNIX sockets (no `hyprctl` subprocess); classic and Lua (≥ 0.55) config parsers; event stream with watch-before-act |
| Screenshot | `grim` (`wlr-screencopy`), monitor / window / region / whole-desktop targets, compositor-side downscaling |
| Coordinates | one authoritative mapper: transmitted px → captured px → logical → per-output pointer extent; fractional scale, negative origins, rotation, headless |
| Input | pure-Python Wayland client speaking `zwlr_virtual_pointer_v1` + `zwp_virtual_keyboard_v1` (unprivileged); `wtype` and `ydotool` as optional fallbacks |
| Headless display | `output create headless` + mode/position; staged access model |
| Browser | in-tree CDP client over loopback (no Playwright); semantic navigate/find/click/type/read/snapshot; shared lock with the visual lane |
| Tools | 30 structured tools: shell, filesystem, git, processes, systemd/journal, podman/docker, clipboard |
| Policy | TOML rules, consequence levels, grants with TTL, `never_grantable`, kill switch, human takeover, JSONL audit with redaction, MCP elicitation for confirmations |
| MCP | in-tree stdio JSON-RPC server (protocol 2024-11-05 … 2025-11-25), image content, elicitation |

Runtime dependency: `httpx` (browser lane only). Everything else is the Python standard library.

## Current limitations (0.1.0)

* Hyprland only. The compositor seam (`desktop/backend.py`) is designed for wlroots/KWin/GNOME/X11 backends, none exist yet.
* Monitor-profile daemons (Omarchy's `hyprmoncfg`, `kanshi`) disable a new headless output within seconds; pause them
  for headless mode (`hyprmoncfg unmanage`), or use the `workspace` stage. Measured: 2.4 s.
* Terminal-intent inference is heuristic (see above).
* Confirmations rely on Codex's MCP elicitation support; other MCP hosts without elicitation get fail-closed denials.
* No X11 / XWayland-specific input handling beyond what the compositor provides.

## Development

```bash
git clone https://github.com/novique-ai/astra-linux-agent && cd astra-linux-agent
uv venv && uv pip install -e '.[dev]'
pytest -q            # unit + integration; live_desktop / live_browser / live_codex are opt-in markers
ruff check . && mypy
python -m testsurface.server   # the synthetic GUI benchmark page
```

Architecture and decisions: [docs/architecture.md](docs/architecture.md), [docs/adr/](docs/adr/).
Contributing: [CONTRIBUTING.md](CONTRIBUTING.md). Security reports: [SECURITY.md](SECURITY.md).

## License

Apache-2.0 — see [LICENSE](LICENSE). Chosen for its explicit patent grant and broad compatibility with both
open-source and commercial use.