Desktop Controller
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
Desktop Controller π₯οΈ
A native Windows Computer-Use & Desktop Automation tool for AI Agents (Antigravity, Claude, ChatGPT, Cursor, etc.).
Unlike traditional tutoring overlays that only point at the screen, Desktop Controller gives the AI agent full interactive capabilities to control Windows directly via the Model Context Protocol (MCP) or CLI.
π Key Features
Pixel-Perfect Multi-Monitor Capture:
High-performance capture using
mss.Native Windows DPI Awareness (
user32.SetProcessDPIAware) preventing coordinate drift on 125%, 150%, or 200% displays.Background Desktop Station attachment (
OpenInputDesktop&SetThreadDesktop) ensuring screenshots never fail from subshells or background workers.
Coordinate Grid Superimposition (
--grid):Superimposes labeled 12Γ8 grid cells (e.g.,
A1,C4,L8) with their exact center pixel coordinates(X, Y)directly onto the screenshot.Allows vision models to accurately select buttons, input fields, and menus without coordinate guessing.
Full Input Automation:
Mouse: Single click, double click, right click, smooth move, drag & drop, and wheel scroll.
Keyboard: Plain text typing, individual key presses (
Enter,Tab,Esc), and hotkey combinations (Ctrl+C,Win+R,Alt+Tab).Window Management: Active window detection (HWND, PID, bounds), open window listing, and automatic window focusing by title substring.
Safety Failsafe:
Built-in
PyAutoGUIfailsafe: slam the mouse cursor into any corner of the screen (e.g.0, 0) to instantly abort runaway mouse actions.
Related MCP server: omni-computer-use
π Project Structure
desktop-controller/
βββ capture.py # DPI-aware screen capture + coordinate grid overlay
βββ input_controller.py # Mouse, keyboard, and window management
βββ server.py # FastMCP stdio server exposing agent tools
βββ cli.py # Command-line testing tool
βββ screenshots/ # Auto-saved captures
βββ README.mdπ οΈ CLI Quick Test
You can test any action directly via terminal:
# 1. Capture screen with coordinate grid
python cli.py screenshot --grid
# 2. Get current desktop state (cursor position, screen size, active window)
python cli.py state
# 3. Move mouse to (500, 500)
python cli.py move 500 500
# 4. Click at (500, 500)
python cli.py click 500 500
# 5. Type text
python cli.py type "Hello from AI Agent"
# 6. Press hotkey (e.g. Win + R)
python cli.py hotkey win r
# 7. Focus a window by title
python cli.py focus "chrome"π Antigravity / MCP Integration
Add this entry to your mcp_config.json (~/.gemini/config/mcp_config.json):
"desktop-controller": {
"command": "python",
"args": [
"C:\\path\\to\\desktop-controller\\server.py"
],
"disabled": false
}Available Tools for AI Agent:
desktop_screenshot(with_grid, monitor_index): Captures screen and returns file path + dimensions.desktop_click(x, y, button, clicks): Clicks coordinates.desktop_double_click(x, y): Double clicks coordinates.desktop_right_click(x, y): Right clicks coordinates.desktop_move(x, y): Moves mouse cursor.desktop_drag(start_x, start_y, end_x, end_y): Drags mouse from A to B.desktop_scroll(clicks, x, y): Scrolls mouse wheel.desktop_type(text): Types text string.desktop_press_key(key): Presses a single key.desktop_hotkey(keys): Presses hotkey combination.desktop_get_state(): Returns cursor position, screen dimensions, and active window.desktop_focus_window(title_query): Brings matching window to foreground.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelogβ¦
This server cannot be deployed
Maintenance
Related MCP Connectors
Connect ChatGPT, Claude, Cursor or VS Code to your Windows PC: files, terminal, browser, apps.
Web scraping, browser automation, screenshots and AI data extraction for AI agents via MCP.
Real desktops for AI agents, on machines you choose: screenshot, type, click, hire more agents.
OCR, transcription, file extraction, and image generation for AI agents via MCP.
Related MCP Servers
- AlicenseAqualityDmaintenanceEnables LLM agents to capture screenshots, control mouse/keyboard, and manage windows on desktop platforms, primarily Windows, via an MCP server.161MIT
- AlicenseNot gradedqualityBmaintenanceEnables Windows desktop automation via MCP, allowing AI agents to control mouse, keyboard, and screen capture with the same interface as Anthropic's computer-use tool.64 PyPI8MIT
- AlicenseAqualityBmaintenanceEnables any AI agent to perceive and control a Windows desktop through live screen capture, OCR, and safety-gated mouse/keyboard actions via a local HTTP API and MCP.81610MIT
- AlicenseNot gradedqualityBmaintenanceGives AI agents full control of a Windows computer over MCP through 63 native tools β screenshots and UI trees for vision, mouse/keyboard input, Chrome control via CDP, plus files, terminal, processes, and SSH/SFTP. The agent can look at the screen, click, type, launch apps, run commands, and verify each step with a fresh screenshot.47MIT