Skip to main content
Glama

Desktop Controller πŸ–₯️

A native Windows Computer-Use & Desktop Automation tool for AI Agents (Antigravity, Claude, ChatGPT, Cursor, etc.).

Unlike traditional tutoring overlays that only point at the screen, Desktop Controller gives the AI agent full interactive capabilities to control Windows directly via the Model Context Protocol (MCP) or CLI.


🌟 Key Features

  1. Pixel-Perfect Multi-Monitor Capture:

    • High-performance capture using mss.

    • Native Windows DPI Awareness (user32.SetProcessDPIAware) preventing coordinate drift on 125%, 150%, or 200% displays.

    • Background Desktop Station attachment (OpenInputDesktop & SetThreadDesktop) ensuring screenshots never fail from subshells or background workers.

  2. Coordinate Grid Superimposition (--grid):

    • Superimposes labeled 12Γ—8 grid cells (e.g., A1, C4, L8) with their exact center pixel coordinates (X, Y) directly onto the screenshot.

    • Allows vision models to accurately select buttons, input fields, and menus without coordinate guessing.

  3. Full Input Automation:

    • Mouse: Single click, double click, right click, smooth move, drag & drop, and wheel scroll.

    • Keyboard: Plain text typing, individual key presses (Enter, Tab, Esc), and hotkey combinations (Ctrl+C, Win+R, Alt+Tab).

    • Window Management: Active window detection (HWND, PID, bounds), open window listing, and automatic window focusing by title substring.

  4. Safety Failsafe:

    • Built-in PyAutoGUI failsafe: slam the mouse cursor into any corner of the screen (e.g. 0, 0) to instantly abort runaway mouse actions.


Related MCP server: omni-computer-use

πŸ“ Project Structure

desktop-controller/
β”œβ”€β”€ capture.py            # DPI-aware screen capture + coordinate grid overlay
β”œβ”€β”€ input_controller.py   # Mouse, keyboard, and window management
β”œβ”€β”€ server.py             # FastMCP stdio server exposing agent tools
β”œβ”€β”€ cli.py                # Command-line testing tool
β”œβ”€β”€ screenshots/          # Auto-saved captures
└── README.md

πŸ› οΈ CLI Quick Test

You can test any action directly via terminal:

# 1. Capture screen with coordinate grid
python cli.py screenshot --grid

# 2. Get current desktop state (cursor position, screen size, active window)
python cli.py state

# 3. Move mouse to (500, 500)
python cli.py move 500 500

# 4. Click at (500, 500)
python cli.py click 500 500

# 5. Type text
python cli.py type "Hello from AI Agent"

# 6. Press hotkey (e.g. Win + R)
python cli.py hotkey win r

# 7. Focus a window by title
python cli.py focus "chrome"

πŸ”Œ Antigravity / MCP Integration

Add this entry to your mcp_config.json (~/.gemini/config/mcp_config.json):

"desktop-controller": {
  "command": "python",
  "args": [
    "C:\\path\\to\\desktop-controller\\server.py"
  ],
  "disabled": false
}

Available Tools for AI Agent:

  • desktop_screenshot(with_grid, monitor_index): Captures screen and returns file path + dimensions.

  • desktop_click(x, y, button, clicks): Clicks coordinates.

  • desktop_double_click(x, y): Double clicks coordinates.

  • desktop_right_click(x, y): Right clicks coordinates.

  • desktop_move(x, y): Moves mouse cursor.

  • desktop_drag(start_x, start_y, end_x, end_y): Drags mouse from A to B.

  • desktop_scroll(clicks, x, y): Scrolls mouse wheel.

  • desktop_type(text): Types text string.

  • desktop_press_key(key): Presses a single key.

  • desktop_hotkey(keys): Presses hotkey combination.

  • desktop_get_state(): Returns cursor position, screen dimensions, and active window.

  • desktop_focus_window(title_query): Brings matching window to foreground.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      D
      maintenance
      Enables LLM agents to capture screenshots, control mouse/keyboard, and manage windows on desktop platforms, primarily Windows, via an MCP server.
      16
      1
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables Windows desktop automation via MCP, allowing AI agents to control mouse, keyboard, and screen capture with the same interface as Anthropic's computer-use tool.
      64 PyPI
      8
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      Gives AI agents full control of a Windows computer over MCP through 63 native tools β€” screenshots and UI trees for vision, mouse/keyboard input, Chrome control via CDP, plus files, terminal, processes, and SSH/SFTP. The agent can look at the screen, click, type, launch apps, run commands, and verify each step with a fresh screenshot.
      47
      MIT