agy_computer_use.mcp_server
Provides native Windows desktop computer-use and GUI grounding tools designed for Google Antigravity 2.0, enabling capture, Set-of-Marks grounding, Win32 input synthesis (clicks, typing, hotkeys, scrolling), background/ghost desktop interaction, and a visual safety oracle with a Merkle audit trail.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@agy_computer_use.mcp_server capture my screen with marks, then click the Save button"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
AGY Computer Use & GUI Grounding (agy-computer-use v1.1.0)
Status: Production Hardened Release (
v1.1.0). Actively tested on Windows 11 (WinSta0\Defaultand backgroundPrintWindow/ Ghost Desktop sessions) with Google Antigravity 2.0.
Native Windows Desktop Computer-Use, UIA + Set-of-Marks (SoM) Grounding, [0, 1000] Bijective Coordinate Normalization, Win32 SendInput & Focusless PostMessageW Action Synthesis, Window Session Management, and RFC 6962 Visual Safety Oracle for Google Antigravity 2.0.
Key Capabilities
Win32 GDI & Synthetic Screen Capture (
agy_computer_use.capture)Live Win32 GDI (
user32/gdi32BitBlt+GetDIBits) desktop and region capture with automaticWinSta0\Defaultsandbox bridge and deterministic synthetic fallback for headless CI.Zero-dependency pure-Python 24-bit BMP (
b"BM") and zlib-compressed PNG (IHDR/IDAT/IEND) encoders.Top-level window enumeration (
enumerate_visible_windows) and interactive child UIA control extraction (extract_uia_element_tree).
Per-Monitor DPI & Set-of-Marks Grounding (
agy_computer_use.som)Per-Monitor DPI V2 scale detection (
get_monitor_dpi_scale).Bijective
[0, 1000] x [0, 1000]coordinate normalization (normalize_to_grid_1000) and denormalization (denormalize_from_grid_1000) with multi-monitor negative origin offset invariance.Visual Set-of-Marks (
annotate_som_elements) bounding box and numbered tag overlay.
GUI Action Dispatch, Clipboard Fast-Path & Fail-Safe Corner (
agy_computer_use.actions)Synthesizes Win32
INPUT(MOUSEINPUT/KEYBDINPUT) structures for 9 action types:click,double_click,right_click,move,drag,scroll,type_text,hotkey,key_tap.Verified
CF_UNICODETEXTclipboard paste (clipboard_paste_text) with read-back verification and prior clipboard state restoration.Direct Set-of-Marks
mark_idtarget lookup and(0, 0)top-left fail-safe abort (ComputerUseSafetyAbort).
Background Window Capture & Ghost Desktop Isolation (
agy_computer_use.background)Captures occluded or off-screen windows without stealing foreground focus via
PrintWindow(PW_RENDERFULLCONTENT)(capture_window_background).Dispatches focusless pointer and keyboard messages via
PostMessageW(WM_LBUTTONDOWN,WM_MOUSEMOVE,WM_CHAR,WM_SETTEXT) (dispatch_background_window_action).Off-screen window staging (
stage_window_offscreen/restore_window_onscreen) and isolated Win32 secondary desktop (GhostDesktopSessionviaCreateDesktopW).
Window Session Manager, UIA Diffing & Bezier Trajectories (
agy_computer_use.window_session)WindowSessionManagerenforcingscreenshot_idgeometry validation (rejecting stale screenshots when target windows move or resize) andCompactionImageBudgetsliding-window image eviction.Hierarchical
#1..#NUIA tree text formatter (format_uia_tree_text) and incremental structural tree diffing (diff_uia_trees).7-control-point (
P0..P6) two-segment cubic Bezier cursor trajectory synthesis (synthesize_bezier_cursor_path),WDA_EXCLUDEFROMCAPTURE(0x11) overlay cloaking (apply_exclude_from_capture), and physical ESC / mouse movementInterruptionMonitor.
64-Bit Perceptual dHash, Prompt-Injection Shield & RFC 6962 Merkle Safety Oracle (
agy_computer_use.safety_oracle)64-bit horizontal gradient difference hash (
compute_frame_dhash64) and Hamming delta classifier ([FACT]for>= 2bits,[NO_EFFECT]for< 2bits).SafetyPolicywindow filter blocking password managers, credential dialogs, UAC, and SmartScreen, plus indirect UI prompt-injection scanner (detect_ui_prompt_injection) and<node_repl_review_evidence>XML isolation (wrap_untrusted_ui_evidence).VisualActionOraclemaintaining an RFC 6962 (0x00leaf /0x01internal node) SHA-256 Merkle audit trail with cryptographic inclusion proof verification (verify_action_receipt).
10-Tool MCP JSON-RPC 2.0 Server (
agy_computer_use.mcp_server)cu_capture_som,cu_click,cu_type,cu_hotkey,cu_scroll,cu_verify_oracle,cu_merkle_audit,cu_bg_capture,cu_bg_dispatch,cu_ghost_desktop(all composite tool identifiers strictly<= 50chars, well below the 64-character Antigravity limit).
Related MCP server: Computer-Controller-Bridg
Installation & Verification
python -m pip install -e . --no-deps
python -m pytest tests/ -vLicense
MIT License - Copyright (c) 2026 orakulzero
Available Tools
12 toolscu_bg_captureB
Capture occluded or offscreen window surface via PrintWindow(PW_RENDERFULLCONTENT) with WM_PRINT fallback.
Inputs: hwnd (required), desktop_name, fallback_seed, fallback_width, +1 more.
cu_bg_dispatchB
Dispatch focusless mouse/keyboard actions via PostMessageW or SendMessageTimeoutW(WM_SETTEXT).
Inputs: hwnd (required), action_type (required), text, norm_x, +7 more.
cu_capture_somC
Capture screen frame, extract UIA controls, and render Set-of-Marks overlay.
Inputs: width, height, max_elements, synthetic_seed.
cu_clickB
Dispatch mouse click, double_click, right_click, move, or drag on [0..1000] grid.
Inputs: norm_x, norm_y, dry_run, mark_id, +1 more.
cu_find_anchorC
Locate template RGB icon on screen using pure Python SAD or NCC matching.
Inputs: width (required), height (required), tmpl_width (required), tmpl_height (required), +4 more.
cu_ghost_desktopC
Manage isolated Win32 kernel desktop session (WinSta0\AGY_GhostDesktop) for zero-focus-theft automation.
Inputs: action (required), hwnd, text, norm_x, +11 more.
cu_hotkeyB
Dispatch multi-key chord sequence (e.g. ctrl+shift+s) or single key tap.
Inputs: keys (required), dry_run.
cu_merkle_auditA
Retrieve RFC 6962 SHA-256 Merkle audit root, levels, and verified receipts.
Inputs: no named parameters.
cu_replay_macroC
Replay recorded trajectory with self-healing UIA selector fallback.
Inputs: trajectory (required), dry_run.
cu_scrollC
Dispatch vertical mouse wheel scroll at [0..1000] normalized coordinates.
Inputs: norm_x, norm_y, dry_run, scroll_delta.
cu_typeB
Synthesize Unicode keyboard text entry via Win32 KEYEVENTF_UNICODE.
Inputs: text (required), dry_run.
cu_verify_oracleC
Verify visual frame delta via 64-bit dHash and record RFC 6962 Merkle receipt.
Inputs: norm_x, norm_y, action_type, target_title, +2 more.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 12 tools
Tools target mostly distinct capabilities: capture, foreground input, background dispatch, visual anchoring, macro replay, and audit/verification. The main overlap is between foreground action tools (cu_click, cu_type, cu_hotkey, cu_scroll) and cu_bg_dispatch, but the focusless PostMessage distinction is explicit; cu_merkle_audit and cu_verify_oracle are also closely related but separable.
All names use the consistent cu_ prefix and snake_case, with predictable action-oriented names. Minor noun-like labels such as cu_hotkey do not break the overall pattern.
Twelve tools is well-scoped for a Windows computer-use server. Each tool covers a distinct capability layer such as capture, input dispatch, verification, background execution, or replay.
Core computer-use operations are covered: screen capture, UIA/SOM extraction, clicking, typing, hotkeys, scrolling, background capture/dispatch, template matching, macro replay, and audit receipts. Gaps include no explicit window enumeration/focus management, clipboard operations, or macro recording, but most can be worked around.
Maintenance
Related MCP Connectors
Eyes and hands on real Windows PCs — observe, click, type via Glasswarp API.
Automate 1,000+ services from any MCP-compatible AI agent: build Applets, run actions and queries.
Connect ChatGPT, Claude, Cursor or VS Code to your Windows PC: files, terminal, browser, apps.
Windows computers for AI agents. Office, desktop apps, browser tasks, testing and file transfer.
Related MCP Servers
- FlicenseNot gradedqualityDmaintenanceEnables automation of native Windows desktop applications through screen capture, mouse/keyboard control, and waiting for UI changes, exposing them as MCP tools.1-
- AlicenseNot gradedqualityCmaintenanceEnables AI clients on Windows to control the local mouse, keyboard, and screen understanding via MCP stdio, allowing automated workflows like viewing the screen, locating elements, clicking, typing, and verifying results.1Apache 2.0
- AlicenseNot gradedqualityBmaintenanceEnables agents to control Windows desktop applications through UI Automation and Win32, supporting window management, mouse/keyboard input, element inspection, screenshots, OCR, macro recording, and autonomous multi-app missions. It also provides an HTTP/stdio MCP interface, an operator UI, and optional telemetry and safety controls.MIT
- AlicenseNot gradedqualityBmaintenanceEnables MCP agents to drive Windows applications by targeting individual windows—clicking, typing, scrolling, dragging, and reading accessibility trees or screenshots—without stealing the user's focus, mouse, or keyboard. Runs entirely locally alongside the user, with per-app approval and no cloud account or usage limits.MIT