Skip to main content
Glama
agy_computer_use.mcp_server

agy_computer_use.mcp_server

by orakulzero

AGY Computer Use & GUI Grounding (agy-computer-use v1.1.0)

Status: Production Hardened Release (v1.1.0). Actively tested on Windows 11 (WinSta0\Default and background PrintWindow / Ghost Desktop sessions) with Google Antigravity 2.0.

Native Windows Desktop Computer-Use, UIA + Set-of-Marks (SoM) Grounding, [0, 1000] Bijective Coordinate Normalization, Win32 SendInput & Focusless PostMessageW Action Synthesis, Window Session Management, and RFC 6962 Visual Safety Oracle for Google Antigravity 2.0.

Key Capabilities

  1. Win32 GDI & Synthetic Screen Capture (agy_computer_use.capture)

    • Live Win32 GDI (user32 / gdi32 BitBlt + GetDIBits) desktop and region capture with automatic WinSta0\Default sandbox bridge and deterministic synthetic fallback for headless CI.

    • Zero-dependency pure-Python 24-bit BMP (b"BM") and zlib-compressed PNG (IHDR / IDAT / IEND) encoders.

    • Top-level window enumeration (enumerate_visible_windows) and interactive child UIA control extraction (extract_uia_element_tree).

  2. Per-Monitor DPI & Set-of-Marks Grounding (agy_computer_use.som)

    • Per-Monitor DPI V2 scale detection (get_monitor_dpi_scale).

    • Bijective [0, 1000] x [0, 1000] coordinate normalization (normalize_to_grid_1000) and denormalization (denormalize_from_grid_1000) with multi-monitor negative origin offset invariance.

    • Visual Set-of-Marks (annotate_som_elements) bounding box and numbered tag overlay.

  3. GUI Action Dispatch, Clipboard Fast-Path & Fail-Safe Corner (agy_computer_use.actions)

    • Synthesizes Win32 INPUT (MOUSEINPUT / KEYBDINPUT) structures for 9 action types: click, double_click, right_click, move, drag, scroll, type_text, hotkey, key_tap.

    • Verified CF_UNICODETEXT clipboard paste (clipboard_paste_text) with read-back verification and prior clipboard state restoration.

    • Direct Set-of-Marks mark_id target lookup and (0, 0) top-left fail-safe abort (ComputerUseSafetyAbort).

  4. Background Window Capture & Ghost Desktop Isolation (agy_computer_use.background)

    • Captures occluded or off-screen windows without stealing foreground focus via PrintWindow(PW_RENDERFULLCONTENT) (capture_window_background).

    • Dispatches focusless pointer and keyboard messages via PostMessageW (WM_LBUTTONDOWN, WM_MOUSEMOVE, WM_CHAR, WM_SETTEXT) (dispatch_background_window_action).

    • Off-screen window staging (stage_window_offscreen / restore_window_onscreen) and isolated Win32 secondary desktop (GhostDesktopSession via CreateDesktopW).

  5. Window Session Manager, UIA Diffing & Bezier Trajectories (agy_computer_use.window_session)

    • WindowSessionManager enforcing screenshot_id geometry validation (rejecting stale screenshots when target windows move or resize) and CompactionImageBudget sliding-window image eviction.

    • Hierarchical #1..#N UIA tree text formatter (format_uia_tree_text) and incremental structural tree diffing (diff_uia_trees).

    • 7-control-point (P0..P6) two-segment cubic Bezier cursor trajectory synthesis (synthesize_bezier_cursor_path), WDA_EXCLUDEFROMCAPTURE (0x11) overlay cloaking (apply_exclude_from_capture), and physical ESC / mouse movement InterruptionMonitor.

  6. 64-Bit Perceptual dHash, Prompt-Injection Shield & RFC 6962 Merkle Safety Oracle (agy_computer_use.safety_oracle)

    • 64-bit horizontal gradient difference hash (compute_frame_dhash64) and Hamming delta classifier ([FACT] for >= 2 bits, [NO_EFFECT] for < 2 bits).

    • SafetyPolicy window filter blocking password managers, credential dialogs, UAC, and SmartScreen, plus indirect UI prompt-injection scanner (detect_ui_prompt_injection) and <node_repl_review_evidence> XML isolation (wrap_untrusted_ui_evidence).

    • VisualActionOracle maintaining an RFC 6962 (0x00 leaf / 0x01 internal node) SHA-256 Merkle audit trail with cryptographic inclusion proof verification (verify_action_receipt).

  7. 10-Tool MCP JSON-RPC 2.0 Server (agy_computer_use.mcp_server)

    • cu_capture_som, cu_click, cu_type, cu_hotkey, cu_scroll, cu_verify_oracle, cu_merkle_audit, cu_bg_capture, cu_bg_dispatch, cu_ghost_desktop (all composite tool identifiers strictly <= 50 chars, well below the 64-character Antigravity limit).

Related MCP server: Computer-Controller-Bridg

Installation & Verification

python -m pip install -e . --no-deps
python -m pytest tests/ -v

License

MIT License - Copyright (c) 2026 orakulzero

Available Tools

12 tools
cu_bg_captureB

Capture occluded or offscreen window surface via PrintWindow(PW_RENDERFULLCONTENT) with WM_PRINT fallback.

Inputs: hwnd (required), desktop_name, fallback_seed, fallback_width, +1 more.

cu_bg_dispatchB

Dispatch focusless mouse/keyboard actions via PostMessageW or SendMessageTimeoutW(WM_SETTEXT).

Inputs: hwnd (required), action_type (required), text, norm_x, +7 more.

cu_capture_somC

Capture screen frame, extract UIA controls, and render Set-of-Marks overlay.

Inputs: width, height, max_elements, synthetic_seed.

cu_clickB

Dispatch mouse click, double_click, right_click, move, or drag on [0..1000] grid.

Inputs: norm_x, norm_y, dry_run, mark_id, +1 more.

cu_find_anchorC

Locate template RGB icon on screen using pure Python SAD or NCC matching.

Inputs: width (required), height (required), tmpl_width (required), tmpl_height (required), +4 more.

cu_ghost_desktopC

Manage isolated Win32 kernel desktop session (WinSta0\AGY_GhostDesktop) for zero-focus-theft automation.

Inputs: action (required), hwnd, text, norm_x, +11 more.

cu_hotkeyB

Dispatch multi-key chord sequence (e.g. ctrl+shift+s) or single key tap.

Inputs: keys (required), dry_run.

cu_merkle_auditA

Retrieve RFC 6962 SHA-256 Merkle audit root, levels, and verified receipts.

Inputs: no named parameters.

cu_replay_macroC

Replay recorded trajectory with self-healing UIA selector fallback.

Inputs: trajectory (required), dry_run.

cu_scrollC

Dispatch vertical mouse wheel scroll at [0..1000] normalized coordinates.

Inputs: norm_x, norm_y, dry_run, scroll_delta.

cu_typeB

Synthesize Unicode keyboard text entry via Win32 KEYEVENTF_UNICODE.

Inputs: text (required), dry_run.

cu_verify_oracleC

Verify visual frame delta via 64-bit dHash and record RFC 6962 Merkle receipt.

Inputs: norm_x, norm_y, action_type, target_title, +2 more.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    B3.4/5.0

    Scored across 12 tools

    Disambiguation4/5

    Tools target mostly distinct capabilities: capture, foreground input, background dispatch, visual anchoring, macro replay, and audit/verification. The main overlap is between foreground action tools (cu_click, cu_type, cu_hotkey, cu_scroll) and cu_bg_dispatch, but the focusless PostMessage distinction is explicit; cu_merkle_audit and cu_verify_oracle are also closely related but separable.

    Naming Consistency5/5

    All names use the consistent cu_ prefix and snake_case, with predictable action-oriented names. Minor noun-like labels such as cu_hotkey do not break the overall pattern.

    Tool Count5/5

    Twelve tools is well-scoped for a Windows computer-use server. Each tool covers a distinct capability layer such as capture, input dispatch, verification, background execution, or replay.

    Completeness4/5

    Core computer-use operations are covered: screen capture, UIA/SOM extraction, clicking, typing, hotkeys, scrolling, background capture/dispatch, template matching, macro replay, and audit receipts. Gaps include no explicit window enumeration/focus management, clipboard operations, or macro recording, but most can be worked around.

    Maintenance

    ActivityMaintained
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers

    • F
      license
      Not graded
      quality
      D
      maintenance
      Enables automation of native Windows desktop applications through screen capture, mouse/keyboard control, and waiting for UI changes, exposing them as MCP tools.
      1
      -
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables AI clients on Windows to control the local mouse, keyboard, and screen understanding via MCP stdio, allowing automated workflows like viewing the screen, locating elements, clicking, typing, and verifying results.
      1
      Apache 2.0
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables agents to control Windows desktop applications through UI Automation and Win32, supporting window management, mouse/keyboard input, element inspection, screenshots, OCR, macro recording, and autonomous multi-app missions. It also provides an HTTP/stdio MCP interface, an operator UI, and optional telemetry and safety controls.
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables MCP agents to drive Windows applications by targeting individual windows—clicking, typing, scrolling, dragging, and reading accessibility trees or screenshots—without stealing the user's focus, mouse, or keyboard. Runs entirely locally alongside the user, with per-app approval and no cloud account or usage limits.
      MIT