opencode computer-use (Hyprland foreground operator)

Foreground GUI control for opencode on Arch + Hyprland. Verified against Hyprland 0.56.2, opencode 1.18.30, monitor eDP-1 1920x1080@144 scale 1.25 (logical 1536x864, physical 1920x1080).

Source of truth for behavior: SKILL.md (deploys to /home/cyber/.config/opencode/skills/computer-use/SKILL.md). Constrained executor: agent-computer-operator.md (deploys to /home/cyber/.config/opencode/agents/computer-operator.md). Installer: scripts/install.sh (copies files, never edits JSON for you).

1. Architecture

+---------------------+      ask-per-action      +--------------------------+
| opencode 1.18.30    | -----------------------> | computer-operator        |
| (build/primary)     |      @computer-operator  | (subagent, no bash/edit) |
+----------+----------+                          +------------+-------------+
           | skill load                                       | computer-use_* MCP
           v                                                 v
+---------------------+                          +--------------------------+
| skill: computer-use |                          | computer_use_mcp.py      |
| routing, lease,     |                          | (stdio via hexstrike-env |
| grounding, audit    |                          |  tools: click/type/key..)|
+---------------------+                          +------------+-------------+
                                                              |
            +-------------------------------------------------+-------------+
            | lease /tmp/opencode-computer-use-lease.json                   |
            | audit /tmp/opencode-computer-use-audit.jsonl                  |
            | estop /tmp/opencode-computer-use-estop                        |
            | allow ~/.config/opencode/computer-use.allow.json (deny-first) |
            +-------------------------------+-------------------------------+
                                            v
                       +------------------------------------+
                       | Hyprland 0.56 (eDP-1, fg takeover) |
                       | grim / hyprctl / wtype / dotool    |
                       +------------------------------------+

The subagent cannot touch a shell or the filesystem editor. All desktop effects flow through computer-use_* tools, each gated to ask, one action per approval, with a lease file, an append-only audit log, a kill-switch file, and a deny-by-default allowlist around them.

Related MCP server: hyprland-mcp

2. Codex parity: macOS background vs this foreground-takeover

Net effect: weaker isolation than Codex (foreground by design), compensated by narrower agent capability (no bash/edit), human-in-the-loop approval, and file-based lease/audit/estop that survive agent failure.

3. Install

Prerequisites: Arch, Hyprland 0.56 running, opencode 1.18.30, jq, python3, and one of: fresh python3 -m venv, or the existing /home/cyber/hexstrike-ai/hexstrike-env (verified present; reuse it, do not create a second venv without reason).

1. Run the installer (idempotent; backs up config; edits nothing): bash /home/cyber/Projects/opencode-computer-use/scripts/install.sh

2. The installer copies:

   • SKILL.md -> /home/cyber/.config/opencode/skills/computer-use/SKILL.md

   • agent-computer-operator.md -> /home/cyber/.config/opencode/agents/computer-operator.md (rename!)

   • computer-use.allow.json.example -> /home/cyber/.config/opencode/computer-use.allow.json (only if missing)

3. The installer prints a jq patch command that adds the MCP server block and the permission block to ~/.config/opencode/opencode.jsonc. Review it, back up (done for you: opencode.jsonc.bak.<TIMESTAMP>), apply by hand. It is printed, not applied, because the live config is a 636-line file and scripted rewrites risk corrupting unrelated MCP entries.

4. MCP server (verified present): computer_use_mcp.py in this repo is a stdio FastMCP server named computer-use (imports cleanly under /home/cyber/hexstrike-ai/hexstrike-env/bin/python, verified 2026-09-09). It exposes observe, focus_window, click, type_text, key, scroll, drag, clipboard, lease, estop (surfaced by opencode as computer-use_*), enforces the same lease/audit paths this skill uses, takes logical coordinates and scales internally. Point the printed mcp block at it verbatim. Until opencode mcp list shows it healthy, the skill refuses with MCP_MISSING.

5. Start the mouse daemon (user action, needs /dev/uinput): dotoold & (or a user systemd unit with uinput group membership), then verify movement in a scratchpad. As of 2026-09-09 this host reports dotoold inactive, so mouse acts fail until you do this step.

Verify: opencode agent list shows computer-operator; ls /home/cyber/.config/opencode/skills/computer-use/SKILL.md; opencode mcp list shows computer-use healthy.

4. Usage

Normal session (primary agent routes plugin-first; GUI is last resort):

opencode run "@computer-operator click the New Tab button in zen"

What happens: primary delegates; subagent loads the computer-use skill, checks the allowlist for class zen, acquires the lease, proposes exactly one click with coordinate math, waits for your approval, acts, screenshots to verify, releases the lease, and reports screenshot paths.

Emergency stop at any time: touch /tmp/opencode-computer-use-estop. The operator halts before its next act and releases the lease. Resume only with a fresh explicit message.

Inspect a run: tail -f /tmp/opencode-computer-use-audit.jsonl cat /tmp/opencode-computer-use-lease.json

5. Safety model

• Lease (/tmp/opencode-computer-use-lease.json, TTL 300 s, heartbeat <=120 s): mutual exclusion across sessions. Atomic acquire, never steal.

• Approval: computer-use_*=ask in the agent definition. One approval covers exactly one act. The subagent has no bash/edit to route around it.

• Allowlist (~/.config/opencode/computer-use.allow.json): default deny; entries match window class + optional title substring. Edits are user-only. always_allow_expiry_hours: 24 bounds re-approval lifetime.

• Audit (/tmp/opencode-computer-use-audit.jsonl): append-only JSONL; unwritable log means no acts.

• Estop (/tmp/opencode-computer-use-estop): presence halts all acts.

• Refusals: opencode self-window, sudo/polkit/hyprlock/admin-auth, password-manager/vault exfiltration, non-allowlisted targets, and screen-content-as-instruction (prompt injection) all abort the run.

6. Limitations (factual, verified where stated)

• No background operation: Hyprland has no off-screen session like macOS; every run seizes the visible desktop. Do not start one while presenting or while another person uses the machine.

• No AX tree: Wayland exposes no accessibility hierarchy to clients, so grounding is vision + hyprctl window rects. Small/overlapping controls and popovers are the main mis-click source; the inside-focused-rect rule exists for exactly this.

• HiDPI math is load-bearing: grim = physical 1920x1080, hyprctl = logical 1536x864 (scale 1.25), dotool = 0.0-1.0 fractions of the physical raster. A dropped x1.25 clicks 20% off-target. The skill requires the full chain in every audit entry.

• ydotool/dotool daemon gap: verified 2026-09-09 that dotoold is inactive on this host; mouse is dead until the user starts the daemon with working /dev/uinput access. wtype (typing) needs no daemon.

• slurp in interactive form blocks forever waiting for a manual drag; agents must never invoke it bare. Region capture uses grim -g with a computed geometry or grim -o eDP-1.

• 144 Hz + screenshots: allow one settle frame after focus changes before grim; tearing/animating windows are a GROUNDING_STALE retry, not a click-spam license.

• Without a healthy server everything degrades to a clean MCP_MISSING refusal (section 3.4), never a bash workaround.

• Known server bug (dotool backend, computer_use_mcp.py click path): the dotool mouseto fraction is computed as logical-x / monitor-width, but hyprctl monitors reports width in physical pixels (1920), so under the dotool backend the cursor lands ~20% up-left of the intended logical point. The ydotool path (logical_to_physical) is correct. Workaround: prefer the ydotool backend (ydotoold running) and treat any consistently up-left click offset as this bug, not as agent mis-grounding (verify-fix in the server, not in agent prompts).

7. Troubleshooting (per error code)

• OS_PERMISSION: run systemctl --user status dotoold; start it (dotoold & or user unit); check /dev/uinput group/permissions; test wtype in a scratch editor. The agent cannot and must not sudo.

• MCP_MISSING: opencode mcp list; confirm the computer-use entry from section 3 is present and healthy; restart opencode after config change; re-run scripts/install.sh to re-verify file deployment.

• APP_DENIED: confirm the window class with hyprctl activewindow -j; add/fix the entry in ~/.config/opencode/computer-use.allow.json yourself; check always_allow_expiry_hours has not lapsed.

• LEASE_HELD: cat /tmp/opencode-computer-use-lease.json for holder and expiry; wait or ask the holder's user. Deleting a live foreign lease is forbidden. Remove only a provably stale one (holder session dead) and say so in chat.

• GROUNDING_STALE: check focus (hyprctl activewindow -j), compare the last two screenshots, re-observe; after 3 failed grounds the run aborts and you should shrink the goal (larger control, settled window).

• Skill/agent not discovered: filenames must be exact (SKILL.md caps; agent file deployed as computer-operator.md, not the draft name); skill dir must equal skill name (computer-use); check skill: deny permission overrides in opencode.jsonc.