# Desktop Controller 🖥️

A native Windows Computer-Use & Desktop Automation tool for AI Agents (Antigravity, Claude, ChatGPT, Cursor, etc.).

Unlike traditional tutoring overlays that only point at the screen, **Desktop Controller** gives the AI agent full interactive capabilities to control Windows directly via the Model Context Protocol (MCP) or CLI.

---

## 🌟 Key Features

1. **Pixel-Perfect Multi-Monitor Capture**:
   - High-performance capture using `mss`.
   - Native Windows DPI Awareness (`user32.SetProcessDPIAware`) preventing coordinate drift on 125%, 150%, or 200% displays.
   - Background Desktop Station attachment (`OpenInputDesktop` & `SetThreadDesktop`) ensuring screenshots never fail from subshells or background workers.

2. **Coordinate Grid Superimposition (`--grid`)**:
   - Superimposes labeled 12×8 grid cells (e.g., `A1`, `C4`, `L8`) with their exact center pixel coordinates `(X, Y)` directly onto the screenshot.
   - Allows vision models to accurately select buttons, input fields, and menus without coordinate guessing.

3. **Full Input Automation**:
   - **Mouse**: Single click, double click, right click, smooth move, drag & drop, and wheel scroll.
   - **Keyboard**: Plain text typing, individual key presses (`Enter`, `Tab`, `Esc`), and hotkey combinations (`Ctrl+C`, `Win+R`, `Alt+Tab`).
   - **Window Management**: Active window detection (HWND, PID, bounds), open window listing, and automatic window focusing by title substring.

4. **Safety Failsafe**:
   - Built-in `PyAutoGUI` failsafe: slam the mouse cursor into any corner of the screen (e.g. `0, 0`) to instantly abort runaway mouse actions.

---

## 📁 Project Structure

```
desktop-controller/
├── capture.py            # DPI-aware screen capture + coordinate grid overlay
├── input_controller.py   # Mouse, keyboard, and window management
├── server.py             # FastMCP stdio server exposing agent tools
├── cli.py                # Command-line testing tool
├── screenshots/          # Auto-saved captures
└── README.md
```

---

## 🛠️ CLI Quick Test

You can test any action directly via terminal:

```bash
# 1. Capture screen with coordinate grid
python cli.py screenshot --grid

# 2. Get current desktop state (cursor position, screen size, active window)
python cli.py state

# 3. Move mouse to (500, 500)
python cli.py move 500 500

# 4. Click at (500, 500)
python cli.py click 500 500

# 5. Type text
python cli.py type "Hello from AI Agent"

# 6. Press hotkey (e.g. Win + R)
python cli.py hotkey win r

# 7. Focus a window by title
python cli.py focus "chrome"
```

---

## 🔌 Antigravity / MCP Integration

Add this entry to your `mcp_config.json` (`~/.gemini/config/mcp_config.json`):

```json
"desktop-controller": {
  "command": "python",
  "args": [
    "C:\\path\\to\\desktop-controller\\server.py"
  ],
  "disabled": false
}
```

### Available Tools for AI Agent:
- `desktop_screenshot(with_grid, monitor_index)`: Captures screen and returns file path + dimensions.
- `desktop_click(x, y, button, clicks)`: Clicks coordinates.
- `desktop_double_click(x, y)`: Double clicks coordinates.
- `desktop_right_click(x, y)`: Right clicks coordinates.
- `desktop_move(x, y)`: Moves mouse cursor.
- `desktop_drag(start_x, start_y, end_x, end_y)`: Drags mouse from A to B.
- `desktop_scroll(clicks, x, y)`: Scrolls mouse wheel.
- `desktop_type(text)`: Types text string.
- `desktop_press_key(key)`: Presses a single key.
- `desktop_hotkey(keys)`: Presses hotkey combination.
- `desktop_get_state()`: Returns cursor position, screen dimensions, and active window.
- `desktop_focus_window(title_query)`: Brings matching window to foreground.