Skip to main content
Glama

PyMCPAutoGUI

by kitfactory

PyMCPAutoGUI 🖱️⌨️🖼️ - GUI Automation via MCP

Supercharge your AI Agent's capabilities! ✨ PyMCPAutoGUI provides a bridge between your AI agents (like those in Cursor or other MCP-compatible environments) and your computer's graphical user interface (GUI). It allows your agent to see the screen 👁️, control the mouse 🖱️ and keyboard ⌨️, and interact with windows 🪟, just like a human user!

Stop tedious manual GUI tasks and let your AI do the heavy lifting 💪. Perfect for automating repetitive actions, testing GUIs, or building powerful AI assistants 🤖.

🤔 Why Choose PyMCPAutoGUI?

  • 🤖 Empower Your Agents: Give your AI agents the power to interact directly with desktop applications.
  • ✅ Simple Integration: Works seamlessly with MCP-compatible clients like the Cursor editor. It's plug and play!
  • 🚀 Easy to Use: Get started with a simple server command. Seriously, it's that easy.
  • 🖱️⌨️ Comprehensive Control: Offers a wide range of GUI automation functions from the battle-tested PyAutoGUI and PyGetWindow.
  • 🖼️ Screen Perception: Includes tools for taking screenshots and locating images on the screen – let your agent see!
  • 🪟 Window Management: Control window position, size, state (minimize, maximize), and more. Tidy up that desktop!
  • 💬 User Interaction: Display alert, confirmation, and prompt boxes to communicate with the user.

🛠️ Supported Environments

  • Operating Systems: Windows, macOS, Linux (Requires appropriate dependencies for pyautogui on each OS)
  • Python: 3.11+ 🐍
  • MCP Clients: Cursor Editor, any client supporting the Model Context Protocol (MCP)

🚀 Getting Started - It's Super Easy!

Using a virtual environment keeps your project dependencies tidy.

# Create and activate a virtual environment (example using venv) python -m venv .venv # Windows PowerShell .venv\Scripts\Activate.ps1 # macOS / Linux bash source .venv/bin/activate # Install using pip (from PyPI or local source) # Make sure your virtual environment is active! pip install pymcpautogui # Or pip install . if installing from local source

(Note: pyautogui might have system dependencies like scrot on Linux for screenshots. Please check the pyautogui documentation for OS-specific installation requirements.)

2. Running the MCP Server

Once installed, simply run the server from your terminal:

# Make sure your virtual environment is activated! python -m pymcpautogui.server

The server will start and listen for connections (defaulting to port 6789). Look for this output:

INFO: Started server process [XXXXX] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://127.0.0.1:6789 (Press CTRL+C to quit)

Keep this terminal running while you need the GUI automation magic! ✨

✨ Seamless Integration with Cursor Editor

Connect PyMCPAutoGUI to Cursor (@ symbol) for GUI automation directly within your coding workflow.

  1. Open MCP Configuration: In Cursor, use the Command Palette (Ctrl+Shift+P or Cmd+Shift+P) and find "MCP: Open mcp.json configuration file".
  2. Add PyMCPAutoGUI Config: Add or merge this configuration into your mcp.json. Adjust paths if needed (especially if Cursor isn't running from the project root).
    { "mcpServers": { // ... other MCP server configs if any ... "PyMCPAutoGUI": { // Sets the working directory. ${workspaceFolder} is usually correct. "cwd": "${workspaceFolder}", // Command to run Python. 'python' works if the venv is active in the terminal // where Cursor was launched, or specify the full path. "command": "python", // Or ".venv/Scripts/python.exe" (Win) or ".venv/bin/python" (Mac/Linux) // Arguments to start the server module. "args": ["-m", "pymcpautogui.server"] } // ... other MCP server configs if any ... } }
    (Tip: If mcp.json already exists, just add the "PyMCPAutoGUI": { ... } part inside the mcpServers object.)
  3. Save mcp.json. Cursor will detect the server.
  4. Automate! Use @PyMCPAutoGUI in Cursor chats:Example: @PyMCPAutoGUI move_to(x=100, y=200) @PyMCPAutoGUI write(text='Automating with AI! 🎉', interval=0.1) @PyMCPAutoGUI screenshot(filename='current_screen.png') @PyMCPAutoGUI activate_window(title='Notepad')

🧰 Available Tools

PyMCPAutoGUI exposes most functions from pyautogui and pygetwindow. Examples include:

  • Mouse 🖱️: move_to, click, move_rel, drag_to, drag_rel, scroll, mouse_down, mouse_up, get_position
  • Keyboard ⌨️: write, press, key_down, key_up, hotkey
  • Screenshots 🖼️: screenshot, locate_on_screen, locate_center_on_screen
  • Windows 🪟: get_all_titles, get_windows_with_title, get_active_window, activate_window, minimize_window, maximize_window, restore_window, move_window, resize_window, close_window
  • Dialogs 💬: alert, confirm, prompt, password
  • Config ⚙️: set_pause, set_failsafe

For the full list and details, check the pymcpautogui/server.py file or use @PyMCPAutoGUI list_tools in your MCP client.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details. Happy Automating! 😄

-
security - not tested
A
license - permissive license
-
quality - not tested

local-only server

The server can only run on the client's local machine because it depends on local resources.

An MCP server that bridges AI agents with GUI automation capabilities, allowing them to control mouse, keyboard, windows, and take screenshots to interact with desktop applications.

  1. 🤔 Why Choose PyMCPAutoGUI?
    1. 🛠️ Supported Environments
      1. 🚀 Getting Started - It's Super Easy!
        1. Installation (Recommended: Use a Virtual Environment!)
        2. Running the MCP Server
      2. ✨ Seamless Integration with Cursor Editor
        1. 🧰 Available Tools
          1. 📄 License

            Related MCP Servers

            • -
              security
              F
              license
              -
              quality
              An MCP server that allows AI assistants like Claude to execute terminal commands on the user's computer and return the output, functioning like a terminal through AI.
              Last updated -
              7
              Python
              • Apple
            • -
              security
              A
              license
              -
              quality
              An enhanced MCP server that grants AI assistants the ability to execute terminal commands on a user's system with improved security controls, designed for use in controlled environments.
              Last updated -
              Python
              MIT License
              • Apple
            • -
              security
              F
              license
              -
              quality
              A MCP server that allows AI assistants to interact with the browser, including getting page content as markdown, modifying page styles, and searching browser history.
              Last updated -
              5
              TypeScript
            • -
              security
              A
              license
              -
              quality
              MCP server that provides computer control capabilities including mouse movements, keyboard actions, screenshot capture with OCR, and window management through a unified API.
              Last updated -
              4
              Python
              MIT License

            View all related MCP servers

            MCP directory API

            We provide all the information about MCP servers via our MCP API.

            curl -X GET 'https://glama.ai/api/mcp/v1/servers/kitfactory/PyMCPAutoGUI'

            If you have feedback or need assistance with the MCP directory API, please join our Discord server