ApplicationController

An MCP server that exposes GUI automation tools to any MCP-compatible client (e.g. GitHub Copilot desktop app). It uses PyAutoGUI to control the mouse, keyboard, and screen of the local machine.

Tools

Related MCP server: MCPControl

Setup

1. Install dependencies

pip install -r requirements.txt

To enable the find_image_on_screen tool, also install OpenCV:

pip install opencv-python

Alternatively, install everything via the package extras:

pip install ".[image-search]"

2. Add to GitHub Copilot desktop app

In your Copilot settings, add an MCP server entry pointing to this server:

{
  "mcpServers": {
    "application-controller": {
      "command": "python",
      "args": ["C:\\path\\to\\ApplicationController\\server.py"]
    }
  }
}

Replace C:\\path\\to\\ApplicationController with the actual path where you cloned this repo.

3. Safety note

PyAutoGUI's failsafe is enabled — move your mouse to the top-left corner of the screen at any time to abort an automation sequence.