Faster Mac Use (fmu)

Background-first, multi-cursor browser and native macOS computer-use engine for AI agents, modeled after and extending OpenAI Codex's cua_repl architecture.

• Website & Interactive Benchmark Explorer: https://lifan-builds.github.io/faster-mac-use/ (docs/index.html)

• Architecture & 5-Workload Benchmark Report: docs/architecture_and_benchmarks.md

• Market Comparison & Positioning Brief: docs/gtm.md

Why faster-mac-use?

Most computer-use tools force a painful choice:

• Pixel screenshot loops (computer tool) hijack your physical mouse cursor, steal window focus, and burn 40k–80k tokens across 10–30 turns.

• Standard Accessibility/Playwright MCPs (@playwright/mcp, chrome-devtools-mcp) inject 4,000–12,500 tokens of tool schemas upfront and return full ARIA trees on every click.

• Single-session REPLs (cua_repl) are fast for one agent, but serialize multi-agent workflows and fall back to screenshots on custom-rendered macOS windows.

faster-mac-use combines a 268-token single-tool MCP/CLI schema, a stateful async cua_repl sandbox, persistent @u1..@uN / @a1..@aN element UIDs with pre-registered table folding, churn-free +/~/- diffs, stale-observation masking, and true multi-cursor parallel execution across unfocused Chrome "🤖 Agent" tab groups and native macOS Accessibility + Apple Silicon Vision.framework OCR — verified at 0.0 px physical cursor movement and 0 focus changes.

Related MCP server: macbot-mcp

Core Architecture

1. Tier 1 — Native Browser Engine (BrowserEngine)

   • Drives Chrome in unfocused, per-agent color-coded "🤖 Agent" Tab Groups (active: false) via extension/background.js and a thread-safe Playwright worker (fmu-pw-worker) that avoids Chrome 136+/144+'s default-profile DevToolsActivePort modal popup.

   • Enables per-tab background focus and lifecycle emulation (Emulation.setFocusEmulationEnabled({enabled: true}) + Page.setWebLifecycleState({state: "active"})) so document.hasFocus() === true, requestAnimationFrame stays at 60fps, and focus-dependent dropdowns work while you work elsewhere.

   • Assigns persistent in-page WeakMap/WeakRef element identities (@u1..@uN) with Pre-Registered Table Row Folding (foldTables), indexing all rows in uidToRef so agents can target folded rows by label or UID while cutting initial dense table snapshots by >2.5x.

   • Emits per-agent compact +/~/- UID deltas with CRC-32 unchanged short-circuiting (= unchanged (crc=...)) and detects + executes WebMCP (navigator.modelContext) tools directly (tab.callTool(name, args)).

   • Intercepts <input type="file"> uploads via CDP (DOM.setFileInputFiles) in exact step sequence so native macOS NSOpenPanel dialogs never steal focus.

2. Tier 2 — Native macOS Accessibility & Vision OCR Fallback (fmu_ax.swift / MacNativeEngine)

   • Concurrent, thread-safe Unix-domain socket daemon (/tmp/fmu-ax.sock, binary bin/fmu-ax) keeping a warm 4,000-element AXUIElement LRU cache, PID/window cache, per-agent +/~/- diff baselines, and per-agent virtual cursors (VirtualCursorState) across turns.

   • Batches accessibility attribute fetches with AXUIElementCopyMultipleAttributeValues and skips redundant pre-action walks when baseline state is already cached.

   • Lazy accessibility wakeup (AXManualAccessibility = true / AXEnhancedUserInterface = true) only for target background PIDs.

   • Stable @a1..@aN UIDs (CFHash + structural fallback), +/~/- UID diffs, and zero-focus background actions (AXPress $\rightarrow$ AXConfirm $\rightarrow$ background CGEvent.postToPid click fallback, plus 16-char UTF-16 batched CGEvent.postToPid keystrokes).

   • Built-in Apple Silicon Vision.framework (VNRecognizeTextRequest) + CGWindowListCreateImage OCR fallback (op: "ocr" / app.ocr()) for GPU/canvas windows with 0 LLM vision tokens.

3. Tier 3 — Unified Multi-Cursor / Multi-Agent Stateful REPL, Warm Daemon & Context Hygiene (FasterMacUseRuntime)

   • Compact single-tool schema (268 tokens) supporting OpenAI Codex-style stateful cua_repl scripts (cua.createBrowserTab, cua.getTab, cua.getApp, cua.getState, cua.getCursors), 1-trip declarative steps batches, and concurrent multi-agent swarms ({"parallel": [{"agent": "agent-1", "code": "..."}, ...]}).

   • Per-agent isolated AgentSession (agent / cursor parameter) with independent REPL handle namespaces, independent ContextWindowTracker stale-snapshot masking, and independent virtual cursor (x, y) coordinates (tab.moveCursor / app.moveCursor).

   • Concurrent persistent runtime daemon (/tmp/fmu-runtime.sock) keeps Playwright browser tabs, WeakMap UIDs, and cua_repl handles warm across parallel CLI/MCP invocations (~79–125 ms for a 10-action web workflow; ~256 ms for a 3-agent 25-action parallel swarm).

   • BackgroundIntrusionGuard polls CoreGraphics (CGEventGetLocation + CGEventSourceSecondsSinceLastEventType) and lsappinfo front at 50Hz to prove 0.0 px cursor displacement and 0 automated focus changes.

Quick Start

1. Install & Build the Native Swift Daemon

git clone https://github.com/lifan-builds/faster-mac-use.git
cd faster-mac-use
pip install -e .
swiftc -O src/native/fmu_ax.swift -o bin/fmu-ax

(Note: if bin/fmu-ax is not pre-built, MacNativeEngine automatically compiles src/native/fmu_ax.swift on first run.)

2. Add to Your MCP Client (Claude Code, Codex, Cursor, Antigravity)

{
  "mcpServers": {
    "faster-mac-use": {
      "command": "/path/to/faster-mac-use/bin/fmu",
      "args": ["mcp"]
    }
  }
}

3. CLI Usage (./bin/fmu)

# Native macOS Calculator in background (zero focus steal)
./bin/fmu '{"surface":"app","app":"Calculator","steps":[{"op":"key","name":"escape"},{"op":"click","name":"7"},{"op":"click","name":"Multiply"},{"op":"click","name":"6"},{"op":"click","name":"Equals"}]}'

# Codex-style stateful cua_repl script
./bin/fmu '{"code":"let tab = await cua.createBrowserTab(\"chrome\", \"https://example.com\", {sessionName: \"🤖 Agent\"});"}'

# Multi-cursor / multi-agent parallel execution across Chrome tabs + native macOS apps
./bin/fmu '{"parallel":[{"agent":"agent-1","code":"let t = await cua.createBrowserTab(\"chrome\", \"https://example.com\"); await t.move(\"More information\");"},{"agent":"agent-2","code":"let a = await cua.getApp(\"Calculator\"); await a.pressKey(\"escape\"); await a.click(\"8\");"}]}'

4. Run Unit & Integration Tests

python3 tests/test_fmu.py

5. Run the 5-Workload Competitor Benchmark Suite

PYTHONPATH=src python3 benchmarks/run_benchmarks.py

Results are written to benchmarks/results/latest_benchmark.md and benchmarks/results/latest_benchmark.json.

License

MIT © Lifan Chen