mcp-autogui-multinode

<p align="center"><a href="" target="_blank"><img src="./mcp-autogui-multinode.png?raw=true"></a></p> <p align="center"> <a href="./"><img src="https://img.shields.io/badge/License-MIT-blue.svg" alt="License MIT"></a> </p> [English](./README.md)|[中文](./README_zh.md) ### About An MCP and HTTP server wrapper for PyAutoGUI, enabling LLMs to control your mouse and keyboard. ### Architecture The service supports two deployment architectures: #### LLM -> MCP -> TOOL (Remote Tool Service) This architecture separates the MCP server from the tool service, allowing the MCP server to connect to a remote tool service via HTTP. ```mermaid graph LR LLM[LLM Client] -->|MCP Protocol| MCP[MCP Server<br/>main.py<br/>Client-based Tools] MCP -->|HTTP API<br/>with API Key| TOOLA[Tool Service<br/>tool.py<br/>HTTP API Server] MCP -->|HTTP API<br/>with API Key| TOOLB[Tool Service<br/>tool.py<br/>HTTP API Server] TOOLA -->|PyAutoGUI| COMPUTERA[Computer Control] TOOLB -->|PyAutoGUI| COMPUTERB[Computer Control] style LLM fill:#e1f5ff style MCP fill:#fff4e1 style TOOLA fill:#ffe1f5 style COMPUTERA fill:#e1ffe1 style COMPUTERB fill:#e1ffe1 ``` **Characteristics:** - MCP server uses **client-based tools** (`register_computer_tools_with_client`) - MCP server forwards requests to remote tool service via HTTP - Tool service performs actual computer control operations - Suitable for distributed deployments where MCP server and tool service run on different machines - Requires `endpoint` parameter in MCP tool calls #### Architecture 2: LLM -> MCP (Direct Tools) This architecture uses direct tools where the MCP server directly performs computer control operations. ```mermaid graph LR LLM[LLM Client] -->|MCP Protocol<br/>stdio/http| MCP[MCP Server<br/>mcp_local.py<br/>Direct Tools] MCP -->|PyAutoGUI| COMPUTER[Computer Control] style LLM fill:#e1f5ff style MCP fill:#fff4e1 style COMPUTER fill:#e1ffe1 ``` **Characteristics:** - MCP server uses **direct tools** (`register_computer_tools`) - MCP server directly executes computer control operations - No separate tool service required - Suitable for local deployments where everything runs on the same machine - No `endpoint` parameter needed in MCP tool calls ## Features - 🚀 **Dual Protocol Support**: HTTP REST API and MCP (Model Context Protocol) - 🔐 **API Key Authentication**: Optional API key authentication for service-to-service communication - 🌐 **Multiple MCP Transports**: Support for both HTTP and stdio (Standard Input/Output) transport modes - 🖱️ **Mouse Control**: Move, click, drag, scroll operations - ⌨️ **Keyboard Control**: Press keys, type text, key combinations - 📸 **Screenshot**: Capture screen and get base64-encoded images - 📊 **Screen Info**: Get cursor position and screen resolution - ⚙️ **Configuration Management**: Pydantic Settings with environment variable support - 📝 **Auto Documentation**: Swagger UI for HTTP API - 🔧 **Flexible Deployment**: Run HTTP server or MCP server independently - 📋 **Request Tracing**: Request ID middleware for request tracking - 📝 **Structured Logging**: Loguru-based logging with request ID integration - 🔌 **Remote MCP Support**: Optional HTTP client for remote tool server integration ## Quick Start ### Prerequisites - Python >= 3.12 - `uv` package manager (recommended) ### Installation 1. Clone the repository: ```bash git clone https://github.com/stonehill-2345/mcp-autogui-multinode.git cd mcp-autogui-multinode ``` 2. Install dependencies based on your deployment scenario: #### Local Full Development For local development with all features (GUI control + testing): ```bash uv sync --group gui --group dev ``` #### Deploy MCP Server Only For deploying MCP server that connects to remote tool service (no GUI dependencies needed): ```bash uv sync --no-group gui ``` #### Deploy Tool Service Only For deploying HTTP tool service that performs actual computer control (requires GUI): ```bash uv sync --group gui ``` ### Running the Service The service supports two independent servers: #### 1. Run Tool Service (HTTP API) Starts the HTTP API server for computer control: ```bash uv run python tool.py ``` #### 2. Run MCP Server Starts the MCP server that can connect to remote tool services. The server supports two transport modes: **HTTP Transport Mode:** ```bash uv run python mcp_local.py http ``` **stdio Transport Mode (default):** ```bash uv run python mcp_local.py stdio ``` After starting, you can access: - **HTTP API Documentation**: http://localhost:8000/docs - **Health Check**: http://localhost:8000/health - **MCP Endpoint**: http://localhost:8001/mcp (if using HTTP transport) ## API Endpoints ### Base Endpoints - `GET /` - Root path, returns API information - `GET /health` - Health check endpoint ### Computer Control Endpoints All computer control actions are available at: - `POST /api/computer/{action}` - Execute a computer control action - `GET /api/computer/actions` - List all available actions #### Available Actions | Action | Description | Parameters | |--------|-------------|------------| | `MoveMouse` | Move mouse cursor | `x`, `y` (coordinates) | | `ClickMouse` | Click mouse button | `x`, `y`, `button`, `press`, `release` | | `PressMouse` | Press mouse button (hold) | `x`, `y`, `button` | | `ReleaseMouse` | Release mouse button | `x`, `y`, `button` | | `DragMouse` | Drag mouse from source to target | `source_x`, `source_y`, `target_x`, `target_y` | | `Scroll` | Scroll mouse wheel | `scroll_direction`, `scroll_amount`, `x`, `y` | | `PressKey` | Press keyboard key(s) | `key` (e.g., "enter", "ctrl c") | | `TypeText` | Type text (uses clipboard) | `text` | | `Wait` | Wait for duration | `duration` (milliseconds) | | `TakeScreenshot` | Capture screen | (no parameters) | | `GetCursorPosition` | Get mouse position | (no parameters) | | `GetScreenSize` | Get screen resolution | (no parameters) | ### Example API Usage #### Move Mouse ```bash curl -X POST "http://localhost:8000/api/computer/MoveMouse" \ -H "Content-Type: application/json" \ -H "X-API-Key: your-secret-api-key-here" \ -d '{"x": 100, "y": 200}' ``` #### Click Mouse ```bash curl -X POST "http://localhost:8000/api/computer/ClickMouse" \ -H "Content-Type: application/json" \ -H "X-API-Key: your-secret-api-key-here" \ -d '{"x": 100, "y": 200, "button": "left"}' ``` #### Take Screenshot ```bash curl -X POST "http://localhost:8000/api/computer/TakeScreenshot" \ -H "Content-Type: application/json" \ -H "X-API-Key: your-secret-api-key-here" \ -d '{}' ``` #### Get Cursor Position ```bash curl -X POST "http://localhost:8000/api/computer/GetCursorPosition" \ -H "Content-Type: application/json" \ -H "X-API-Key: your-secret-api-key-here" \ -d '{}' ``` **Note**: If `API_KEY_ENABLED=false`, the `X-API-Key` header is optional. If `API_KEY_ENABLED=true`, the header is required for all requests except health checks and documentation endpoints. ## MCP Tools The service also exposes all computer control operations as MCP tools. When running the MCP server, you can use these tools through any MCP-compatible client. ### Available MCP Tools All HTTP API actions are available as MCP tools. The MCP tool names use snake_case, while the HTTP API uses PascalCase: - `move_mouse` - Move mouse cursor (HTTP: `MoveMouse`) - `click_mouse` - Click mouse button (HTTP: `ClickMouse`) - `press_mouse` - Press mouse button (HTTP: `PressMouse`) - `release_mouse` - Release mouse button (HTTP: `ReleaseMouse`) - `drag_mouse` - Drag mouse (HTTP: `DragMouse`) - `scroll` - Scroll mouse wheel (HTTP: `Scroll`) - `press_key` - Press keyboard key (HTTP: `PressKey`) - `type_text` - Type text (HTTP: `TypeText`) - `wait` - Wait for duration (HTTP: `Wait`) - `take_screenshot` - Take screenshot (HTTP: `TakeScreenshot`) - `get_cursor_position` - Get cursor position (HTTP: `GetCursorPosition`) - `get_screen_size` - Get screen size (HTTP: `GetScreenSize`) ### MCP Transport Modes The MCP server supports two transport modes: 1. **stdio** (default): Standard input/output transport - Used for local communication via stdin/stdout - Suitable for direct integration with MCP clients - Start with: `python mcp_local.py stdio` 2. **http**: HTTP-based transport with stateless mode - Used for remote communication over HTTP - Suitable for service-to-service communication - Start with: `python mcp_local.py http` - Accessible at: `http://localhost:8001/mcp` ### MCP Tool Registration Modes The service supports two modes of MCP tool registration: 1. **Direct Tools** (`register_computer_tools`): Tools that directly call the local computer control implementation. No `endpoint` parameter required. - Used in `mcp_local.py` for local MCP server - Tools execute computer control actions directly 2. **Client-based Tools** (`register_computer_tools_with_client`): Tools that use an HTTP client to call a remote tool server. Requires an `endpoint` parameter. - Used in `mcp_server/register.py` for remote MCP server - Tools forward requests to a remote tool service via HTTP The local MCP server (`mcp_local.py`) uses direct tools by default. The remote MCP server uses client-based tools. ### Code Style - Use type hints for all function parameters and return types - Follow PEP 8 style guidelines - Use descriptive docstrings for all public functions - Keep functions focused and single-purpose ## Security Considerations ⚠️ **Warning**: This service provides direct control over your computer's mouse and keyboard. Use with caution: - Only run on trusted networks - Restrict CORS origins in production (currently allows all origins) - **Enable API Key Authentication**: Set `API_KEY_ENABLED=true` and configure a strong `API_KEY` in production - Be aware of the security implications of remote computer control ### API Key Authentication The service supports optional API key authentication for securing service-to-service communication: 1. **Enable Authentication**: Set `API_KEY_ENABLED=true` in your `.env` file 2. **Set API Key**: Configure `API_KEY=your-secret-api-key-here` in your `.env` file 3. **Pass API Key in Requests**: Include the API key in request headers: - `X-API-Key: your-secret-api-key-here` (recommended) - `Authorization: Bearer your-secret-api-key-here` (alternative) **Excluded Paths** (no authentication required): - `/health` - Health check endpoint - `/docs` - API documentation - `/openapi.json` - OpenAPI schema - `/redoc` - Alternative API documentation **MCP Client Usage with API Key**: ```python from fastmcp import Client from fastmcp.client.transports import StreamableHttpTransport # Create transport with API key transport = StreamableHttpTransport( url="http://localhost:8001/mcp", headers={"X-API-Key": "your-secret-api-key-here"} ) # Create client with transport client = Client(transport) async with client: response = await client.call_tool("move_mouse", {"x": 100, "y": 200}) ``` ## Logging The service uses Loguru for structured logging with the following features: - **Request ID Tracking**: Each request gets a unique ID that appears in all log entries - **Environment-aware**: Console output in development, file logging in production - **Structured Format**: Includes timestamp, level, request ID, module, function, and line number Log files are stored in the `logs/` directory: - `app_YYYY-MM-DD.log`: General application logs - `error_YYYY-MM-DD.log`: Error logs only In development mode, logs are only output to the console. In production mode, logs are written to both console and files. ## Testing Run tests using pytest: ```bash # Run all tests uv pytest # Run specific test file uvpytest tests/test_mcp_client.py # Run with verbose output pytest -v ``` The test suite includes: - `test_local_mcp_client.py`: Tests for local MCP server with HTTP transport (direct tools) - `test_stdio_mcp_client.py`: Tests for local MCP server with stdio transport (direct tools) - `test_mcp_client.py`: Tests for remote MCP server with client-based tools (requires endpoint parameter) ## Troubleshooting ### Port Already in Use If you get a port already in use error: ```bash # Change ports in .env file PORT=8002 MCP_PORT=8003 ``` ### MCP Connection Issues For HTTP transport, ensure the MCP server is running and accessible: ```bash # Test MCP endpoint curl http://localhost:8001/mcp # Test with API key (if enabled) curl -H "X-API-Key: your-secret-api-key-here" http://localhost:8001/mcp ``` For stdio transport, ensure the MCP server is started with stdio mode: ```bash # Start MCP server in stdio mode uv python mcp_local.py stdio ``` ### API Key Authentication Issues If you're getting authentication errors: - Verify `API_KEY_ENABLED` is set correctly in `.env` - Check that `API_KEY` matches between client and server - Ensure the API key is passed in the `X-API-Key` header or `Authorization: Bearer <key>` header - Check that the request path is not in the excluded paths list ### Screenshot Issues If screenshot functionality fails: - Check Python version compatibility (requires Python >= 3.12) - Verify display permissions on macOS/Linux - Ensure PyAutoGUI and its dependencies are properly installed ## License [MIT license](./LICENSE) ## Contributing Contributions are welcome! Please feel free to submit a Pull Request.