# opencode computer-use (Hyprland foreground operator)

Foreground GUI control for opencode on Arch + Hyprland. Verified against
Hyprland 0.56.2, opencode 1.18.30, monitor eDP-1 1920x1080@144 scale 1.25
(logical 1536x864, physical 1920x1080).

Source of truth for behavior: `SKILL.md` (deploys to
`/home/cyber/.config/opencode/skills/computer-use/SKILL.md`).
Constrained executor: `agent-computer-operator.md` (deploys to
`/home/cyber/.config/opencode/agents/computer-operator.md`).
Installer: `scripts/install.sh` (copies files, never edits JSON for you).

## 1. Architecture

```
+---------------------+      ask-per-action      +--------------------------+
| opencode 1.18.30    | -----------------------> | computer-operator        |
| (build/primary)     |      @computer-operator  | (subagent, no bash/edit) |
+----------+----------+                          +------------+-------------+
           | skill load                                       | computer-use_* MCP
           v                                                 v
+---------------------+                          +--------------------------+
| skill: computer-use |                          | computer_use_mcp.py      |
| routing, lease,     |                          | (stdio via hexstrike-env |
| grounding, audit    |                          |  tools: click/type/key..)|
+---------------------+                          +------------+-------------+
                                                              |
            +-------------------------------------------------+-------------+
            | lease /tmp/opencode-computer-use-lease.json                   |
            | audit /tmp/opencode-computer-use-audit.jsonl                  |
            | estop /tmp/opencode-computer-use-estop                        |
            | allow ~/.config/opencode/computer-use.allow.json (deny-first) |
            +-------------------------------+-------------------------------+
                                            v
                       +------------------------------------+
                       | Hyprland 0.56 (eDP-1, fg takeover) |
                       | grim / hyprctl / wtype / dotool    |
                       +------------------------------------+
```

The subagent cannot touch a shell or the filesystem editor. All desktop
effects flow through `computer-use_*` tools, each gated to `ask`, one action
per approval, with a lease file, an append-only audit log, a kill-switch
file, and a deny-by-default allowlist around them.

## 2. Codex parity: macOS background vs this foreground-takeover

| Dimension        | Codex (macOS)                                  | This project (Hyprland)                              |
|------------------|------------------------------------------------|------------------------------------------------------|
| Session model    | Background session, sandboxed                  | Foreground takeover: user watches, screen is seized  |
| Grounding        | Accessibility (AX) tree + screenshot           | Screenshot + `hyprctl` rect only; no AX tree exists  |
| Input path       | OS automation API (CUA)                        | `dotool`/`dotoold` mouse (fractions 0-1), `wtype` keys |
| Concurrency      | Parallel background runs                       | Forbidden: one lease, one target, one action at a time |
| Permissions      | Per-action macOS consent prompts               | `computer-use_*=ask` + allowlist + lease + estop     |
| Audit            | Session transcript                             | `/tmp/opencode-computer-use-audit.jsonl` (JSONL)     |
| Credential guard | Sandbox policy                                 | Hard refusal: sudo/polkit/hyprlock/vault surfaces    |

Net effect: weaker isolation than Codex (foreground by design), compensated
by narrower agent capability (no bash/edit), human-in-the-loop approval, and
file-based lease/audit/estop that survive agent failure.

## 3. Install

Prerequisites: Arch, Hyprland 0.56 running, opencode 1.18.30, `jq`,
`python3`, and one of: fresh `python3 -m venv`, or the existing
`/home/cyber/hexstrike-ai/hexstrike-env` (verified present; reuse it, do not
create a second venv without reason).

1. Run the installer (idempotent; backs up config; edits nothing):
   `bash /home/cyber/Projects/opencode-computer-use/scripts/install.sh`
2. The installer copies:
   - `SKILL.md` -> `/home/cyber/.config/opencode/skills/computer-use/SKILL.md`
   - `agent-computer-operator.md` ->
     `/home/cyber/.config/opencode/agents/computer-operator.md` (rename!)
   - `computer-use.allow.json.example` ->
     `/home/cyber/.config/opencode/computer-use.allow.json` (only if missing)
3. The installer prints a `jq` patch command that adds the MCP server block
   and the permission block to `~/.config/opencode/opencode.jsonc`. Review
   it, back up (done for you: `opencode.jsonc.bak.<TIMESTAMP>`), apply by
   hand. It is printed, not applied, because the live config is a 636-line
   file and scripted rewrites risk corrupting unrelated MCP entries.
4. MCP server (verified present): `computer_use_mcp.py` in this repo is a
   stdio FastMCP server named `computer-use` (imports cleanly under
   `/home/cyber/hexstrike-ai/hexstrike-env/bin/python`, verified
   2026-09-09). It exposes `observe`, `focus_window`, `click`, `type_text`,
   `key`, `scroll`, `drag`, `clipboard`, `lease`, `estop` (surfaced by
   opencode as `computer-use_*`), enforces the same lease/audit paths this
   skill uses, takes **logical** coordinates and scales internally. Point
   the printed `mcp` block at it verbatim. Until `opencode mcp list` shows
   it healthy, the skill refuses with `MCP_MISSING`.
5. Start the mouse daemon (user action, needs `/dev/uinput`):
   `dotoold &` (or a user systemd unit with `uinput` group membership),
   then verify movement in a scratchpad. As of 2026-09-09 this host reports
   `dotoold` inactive, so mouse acts fail until you do this step.

Verify: `opencode agent list` shows `computer-operator`;
`ls /home/cyber/.config/opencode/skills/computer-use/SKILL.md`;
`opencode mcp list` shows `computer-use` healthy.

## 4. Usage

Normal session (primary agent routes plugin-first; GUI is last resort):

```
opencode run "@computer-operator click the New Tab button in zen"
```

What happens: primary delegates; subagent loads the `computer-use` skill,
checks the allowlist for class `zen`, acquires the lease, proposes exactly
one click with coordinate math, waits for your approval, acts, screenshots
to verify, releases the lease, and reports screenshot paths.

Emergency stop at any time: `touch /tmp/opencode-computer-use-estop`.
The operator halts before its next act and releases the lease. Resume only
with a fresh explicit message.

Inspect a run:
`tail -f /tmp/opencode-computer-use-audit.jsonl`
`cat /tmp/opencode-computer-use-lease.json`

## 5. Safety model

- **Lease** (`/tmp/opencode-computer-use-lease.json`, TTL 300 s, heartbeat
  <=120 s): mutual exclusion across sessions. Atomic acquire, never steal.
- **Approval**: `computer-use_*=ask` in the agent definition. One approval
  covers exactly one act. The subagent has no `bash`/`edit` to route around it.
- **Allowlist** (`~/.config/opencode/computer-use.allow.json`): default
  deny; entries match window class + optional title substring. Edits are
  user-only. `always_allow_expiry_hours: 24` bounds re-approval lifetime.
- **Audit** (`/tmp/opencode-computer-use-audit.jsonl`): append-only JSONL;
  unwritable log means no acts.
- **Estop** (`/tmp/opencode-computer-use-estop`): presence halts all acts.
- **Refusals**: opencode self-window, sudo/polkit/hyprlock/admin-auth,
  password-manager/vault exfiltration, non-allowlisted targets, and
  screen-content-as-instruction (prompt injection) all abort the run.

## 6. Limitations (factual, verified where stated)

- No background operation: Hyprland has no off-screen session like macOS;
  every run seizes the visible desktop. Do not start one while presenting
  or while another person uses the machine.
- No AX tree: Wayland exposes no accessibility hierarchy to clients, so
  grounding is vision + `hyprctl` window rects. Small/overlapping controls
  and popovers are the main mis-click source; the inside-focused-rect rule
  exists for exactly this.
- HiDPI math is load-bearing: grim = physical 1920x1080, hyprctl = logical
  1536x864 (scale 1.25), dotool = 0.0-1.0 fractions of the physical raster.
  A dropped x1.25 clicks 20% off-target. The skill requires the full chain
  in every audit entry.
- `ydotool`/`dotool` daemon gap: verified 2026-09-09 that `dotoold` is
  inactive on this host; mouse is dead until the user starts the daemon
  with working `/dev/uinput` access. `wtype` (typing) needs no daemon.
- `slurp` in interactive form blocks forever waiting for a manual drag;
  agents must never invoke it bare. Region capture uses `grim -g` with a
  computed geometry or `grim -o eDP-1`.
- 144 Hz + screenshots: allow one settle frame after focus changes before
  `grim`; tearing/animating windows are a `GROUNDING_STALE` retry, not a
  click-spam license.
- Without a healthy server everything degrades to a clean `MCP_MISSING`
  refusal (section 3.4), never a bash workaround.
- Known server bug (dotool backend, `computer_use_mcp.py` click path): the
  dotool `mouseto` fraction is computed as logical-x / monitor-width, but
  `hyprctl monitors` reports width in physical pixels (1920), so under the
  dotool backend the cursor lands ~20% up-left of the intended logical
  point. The ydotool path (`logical_to_physical`) is correct. Workaround:
  prefer the ydotool backend (`ydotoold` running) and treat any
  consistently up-left click offset as this bug, not as agent mis-grounding
  (verify-fix in the server, not in agent prompts).

## 7. Troubleshooting (per error code)

- `OS_PERMISSION`: run `systemctl --user status dotoold`; start it
  (`dotoold &` or user unit); check `/dev/uinput` group/permissions;
  test `wtype` in a scratch editor. The agent cannot and must not sudo.
- `MCP_MISSING`: `opencode mcp list`; confirm the `computer-use` entry from
  section 3 is present and healthy; restart opencode after config change;
  re-run `scripts/install.sh` to re-verify file deployment.
- `APP_DENIED`: confirm the window class with `hyprctl activewindow -j`;
  add/fix the entry in `~/.config/opencode/computer-use.allow.json`
  yourself; check `always_allow_expiry_hours` has not lapsed.
- `LEASE_HELD`: `cat /tmp/opencode-computer-use-lease.json` for holder and
  expiry; wait or ask the holder's user. Deleting a live foreign lease is
  forbidden. Remove only a provably stale one (holder session dead) and say
  so in chat.
- `GROUNDING_STALE`: check focus (`hyprctl activewindow -j`), compare the
  last two screenshots, re-observe; after 3 failed grounds the run aborts
  and you should shrink the goal (larger control, settled window).
- Skill/agent not discovered: filenames must be exact (`SKILL.md` caps;
  agent file deployed as `computer-operator.md`, not the draft name);
  skill dir must equal skill name (`computer-use`); check
  `skill: deny` permission overrides in `opencode.jsonc`.