Desktop Controller 🖥️

A native Windows Computer-Use & Desktop Automation tool for AI Agents (Antigravity, Claude, ChatGPT, Cursor, etc.).

Unlike traditional tutoring overlays that only point at the screen, Desktop Controller gives the AI agent full interactive capabilities to control Windows directly via the Model Context Protocol (MCP) or CLI.

🌟 Key Features

1. Pixel-Perfect Multi-Monitor Capture:

   • High-performance capture using mss.

   • Native Windows DPI Awareness (user32.SetProcessDPIAware) preventing coordinate drift on 125%, 150%, or 200% displays.

   • Background Desktop Station attachment (OpenInputDesktop & SetThreadDesktop) ensuring screenshots never fail from subshells or background workers.

2. Coordinate Grid Superimposition (--grid):

   • Superimposes labeled 12×8 grid cells (e.g., A1, C4, L8) with their exact center pixel coordinates (X, Y) directly onto the screenshot.

   • Allows vision models to accurately select buttons, input fields, and menus without coordinate guessing.

3. Full Input Automation:

   • Mouse: Single click, double click, right click, smooth move, drag & drop, and wheel scroll.

   • Keyboard: Plain text typing, individual key presses (Enter, Tab, Esc), and hotkey combinations (Ctrl+C, Win+R, Alt+Tab).

   • Window Management: Active window detection (HWND, PID, bounds), open window listing, and automatic window focusing by title substring.

4. Safety Failsafe:

   • Built-in PyAutoGUI failsafe: slam the mouse cursor into any corner of the screen (e.g. 0, 0) to instantly abort runaway mouse actions.

Related MCP server: pov

📁 Project Structure

desktop-controller/
├── capture.py            # DPI-aware screen capture + coordinate grid overlay
├── input_controller.py   # Mouse, keyboard, and window management
├── server.py             # FastMCP stdio server exposing agent tools
├── cli.py                # Command-line testing tool
├── screenshots/          # Auto-saved captures
└── README.md

🛠️ CLI Quick Test

You can test any action directly via terminal:

# 1. Capture screen with coordinate grid
python cli.py screenshot --grid

# 2. Get current desktop state (cursor position, screen size, active window)
python cli.py state

# 3. Move mouse to (500, 500)
python cli.py move 500 500

# 4. Click at (500, 500)
python cli.py click 500 500

# 5. Type text
python cli.py type "Hello from AI Agent"

# 6. Press hotkey (e.g. Win + R)
python cli.py hotkey win r

# 7. Focus a window by title
python cli.py focus "chrome"

🔌 Antigravity / MCP Integration

Add this entry to your mcp_config.json (~/.gemini/config/mcp_config.json):

"desktop-controller": {
  "command": "python",
  "args": [
    "C:\\path\\to\\desktop-controller\\server.py"
  ],
  "disabled": false
}

Available Tools for AI Agent:

• desktop_screenshot(with_grid, monitor_index): Captures screen and returns file path + dimensions.

• desktop_click(x, y, button, clicks): Clicks coordinates.

• desktop_double_click(x, y): Double clicks coordinates.

• desktop_right_click(x, y): Right clicks coordinates.

• desktop_move(x, y): Moves mouse cursor.

• desktop_drag(start_x, start_y, end_x, end_y): Drags mouse from A to B.

• desktop_scroll(clicks, x, y): Scrolls mouse wheel.

• desktop_type(text): Types text string.

• desktop_press_key(key): Presses a single key.

• desktop_hotkey(keys): Presses hotkey combination.

• desktop_get_state(): Returns cursor position, screen dimensions, and active window.

• desktop_focus_window(title_query): Brings matching window to foreground.