Skip to main content
Glama
stonehill-2345

mcp-autogui-multinode

by stonehill-2345
OverviewSchemaRelated ServersScoreDiscussions
Python
Hybrid

English|δΈ­ζ–‡

About

An MCP and HTTP server wrapper for PyAutoGUI, enabling LLMs to control your mouse and keyboard.

Architecture

The service supports two deployment architectures:

LLM -> MCP -> TOOL (Remote Tool Service)

This architecture separates the MCP server from the tool service, allowing the MCP server to connect to a remote tool service via HTTP.

graph LR LLM[LLM Client] -->|MCP Protocol| MCP[MCP Server<br/>main.py<br/>Client-based Tools] MCP -->|HTTP API<br/>with API Key| TOOLA[Tool Service<br/>tool.py<br/>HTTP API Server] MCP -->|HTTP API<br/>with API Key| TOOLB[Tool Service<br/>tool.py<br/>HTTP API Server] TOOLA -->|PyAutoGUI| COMPUTERA[Computer Control] TOOLB -->|PyAutoGUI| COMPUTERB[Computer Control] style LLM fill:#e1f5ff style MCP fill:#fff4e1 style TOOLA fill:#ffe1f5 style COMPUTERA fill:#e1ffe1 style COMPUTERB fill:#e1ffe1

Characteristics:

  • MCP server uses client-based tools (register_computer_tools_with_client)

  • MCP server forwards requests to remote tool service via HTTP

  • Tool service performs actual computer control operations

  • Suitable for distributed deployments where MCP server and tool service run on different machines

  • Requires endpoint parameter in MCP tool calls

Architecture 2: LLM -> MCP (Direct Tools)

This architecture uses direct tools where the MCP server directly performs computer control operations.

graph LR LLM[LLM Client] -->|MCP Protocol<br/>stdio/http| MCP[MCP Server<br/>mcp_local.py<br/>Direct Tools] MCP -->|PyAutoGUI| COMPUTER[Computer Control] style LLM fill:#e1f5ff style MCP fill:#fff4e1 style COMPUTER fill:#e1ffe1

Characteristics:

  • MCP server uses direct tools (register_computer_tools)

  • MCP server directly executes computer control operations

  • No separate tool service required

  • Suitable for local deployments where everything runs on the same machine

  • No endpoint parameter needed in MCP tool calls

Features

  • πŸš€ Dual Protocol Support: HTTP REST API and MCP (Model Context Protocol)

  • πŸ” API Key Authentication: Optional API key authentication for service-to-service communication

  • 🌐 Multiple MCP Transports: Support for both HTTP and stdio (Standard Input/Output) transport modes

  • πŸ–±οΈ Mouse Control: Move, click, drag, scroll operations

  • ⌨️ Keyboard Control: Press keys, type text, key combinations

  • πŸ“Έ Screenshot: Capture screen and get base64-encoded images

  • πŸ“Š Screen Info: Get cursor position and screen resolution

  • βš™οΈ Configuration Management: Pydantic Settings with environment variable support

  • πŸ“ Auto Documentation: Swagger UI for HTTP API

  • πŸ”§ Flexible Deployment: Run HTTP server or MCP server independently

  • πŸ“‹ Request Tracing: Request ID middleware for request tracking

  • πŸ“ Structured Logging: Loguru-based logging with request ID integration

  • πŸ”Œ Remote MCP Support: Optional HTTP client for remote tool server integration

Quick Start

Prerequisites

  • Python >= 3.12

  • uv package manager (recommended)

Installation

  1. Clone the repository:

git clone https://github.com/stonehill-2345/mcp-autogui-multinode.git cd mcp-autogui-multinode

  1. Install dependencies based on your deployment scenario:

Local Full Development

For local development with all features (GUI control + testing):

uv sync --group gui --group dev

Deploy MCP Server Only

For deploying MCP server that connects to remote tool service (no GUI dependencies needed):

uv sync --no-group gui

Deploy Tool Service Only

For deploying HTTP tool service that performs actual computer control (requires GUI):

uv sync --group gui

Running the Service

The service supports two independent servers:

1. Run Tool Service (HTTP API)

Starts the HTTP API server for computer control:

uv run python tool.py

2. Run MCP Server

Starts the MCP server that can connect to remote tool services. The server supports two transport modes:

HTTP Transport Mode:

uv run python mcp_local.py http

stdio Transport Mode (default):

uv run python mcp_local.py stdio

After starting, you can access:

API Endpoints

Base Endpoints

  • GET / - Root path, returns API information

  • GET /health - Health check endpoint

Computer Control Endpoints

All computer control actions are available at:

  • POST /api/computer/{action} - Execute a computer control action

  • GET /api/computer/actions - List all available actions

Available Actions

Action

Description

Parameters

MoveMouse

Move mouse cursor

x, y (coordinates)

ClickMouse

Click mouse button

x, y, button, press, release

PressMouse

Press mouse button (hold)

x, y, button

ReleaseMouse

Release mouse button

x, y, button

DragMouse

Drag mouse from source to target

source_x, source_y, target_x, target_y

Scroll

Scroll mouse wheel

scroll_direction, scroll_amount, x, y

PressKey

Press keyboard key(s)

key (e.g., "enter", "ctrl c")

TypeText

Type text (uses clipboard)

text

Wait

Wait for duration

duration (milliseconds)

TakeScreenshot

Capture screen

(no parameters)

GetCursorPosition

Get mouse position

(no parameters)

GetScreenSize

Get screen resolution

(no parameters)

Example API Usage

Move Mouse

curl -X POST "http://localhost:8000/api/computer/MoveMouse" \ -H "Content-Type: application/json" \ -H "X-API-Key: your-secret-api-key-here" \ -d '{"x": 100, "y": 200}'

Click Mouse

curl -X POST "http://localhost:8000/api/computer/ClickMouse" \ -H "Content-Type: application/json" \ -H "X-API-Key: your-secret-api-key-here" \ -d '{"x": 100, "y": 200, "button": "left"}'

Take Screenshot

curl -X POST "http://localhost:8000/api/computer/TakeScreenshot" \ -H "Content-Type: application/json" \ -H "X-API-Key: your-secret-api-key-here" \ -d '{}'

Get Cursor Position

curl -X POST "http://localhost:8000/api/computer/GetCursorPosition" \ -H "Content-Type: application/json" \ -H "X-API-Key: your-secret-api-key-here" \ -d '{}'

Note: If API_KEY_ENABLED=false, the X-API-Key header is optional. If API_KEY_ENABLED=true, the header is required for all requests except health checks and documentation endpoints.

MCP Tools

The service also exposes all computer control operations as MCP tools. When running the MCP server, you can use these tools through any MCP-compatible client.

Available MCP Tools

All HTTP API actions are available as MCP tools. The MCP tool names use snake_case, while the HTTP API uses PascalCase:

  • move_mouse - Move mouse cursor (HTTP: MoveMouse)

  • click_mouse - Click mouse button (HTTP: ClickMouse)

  • press_mouse - Press mouse button (HTTP: PressMouse)

  • release_mouse - Release mouse button (HTTP: ReleaseMouse)

  • drag_mouse - Drag mouse (HTTP: DragMouse)

  • scroll - Scroll mouse wheel (HTTP: Scroll)

  • press_key - Press keyboard key (HTTP: PressKey)

  • type_text - Type text (HTTP: TypeText)

  • wait - Wait for duration (HTTP: Wait)

  • take_screenshot - Take screenshot (HTTP: TakeScreenshot)

  • get_cursor_position - Get cursor position (HTTP: GetCursorPosition)

  • get_screen_size - Get screen size (HTTP: GetScreenSize)

MCP Transport Modes

The MCP server supports two transport modes:

  1. stdio (default): Standard input/output transport

    • Used for local communication via stdin/stdout

    • Suitable for direct integration with MCP clients

    • Start with: python mcp_local.py stdio

  2. http: HTTP-based transport with stateless mode

    • Used for remote communication over HTTP

    • Suitable for service-to-service communication

    • Start with: python mcp_local.py http

    • Accessible at: http://localhost:8001/mcp

MCP Tool Registration Modes

The service supports two modes of MCP tool registration:

  1. Direct Tools (register_computer_tools): Tools that directly call the local computer control implementation. No endpoint parameter required.

    • Used in mcp_local.py for local MCP server

    • Tools execute computer control actions directly

  2. Client-based Tools (register_computer_tools_with_client): Tools that use an HTTP client to call a remote tool server. Requires an endpoint parameter.

    • Used in mcp_server/register.py for remote MCP server

    • Tools forward requests to a remote tool service via HTTP

The local MCP server (mcp_local.py) uses direct tools by default. The remote MCP server uses client-based tools.

Code Style

  • Use type hints for all function parameters and return types

  • Follow PEP 8 style guidelines

  • Use descriptive docstrings for all public functions

  • Keep functions focused and single-purpose

Security Considerations

⚠️ Warning: This service provides direct control over your computer's mouse and keyboard. Use with caution:

  • Only run on trusted networks

  • Restrict CORS origins in production (currently allows all origins)

  • Enable API Key Authentication: Set API_KEY_ENABLED=true and configure a strong API_KEY in production

  • Be aware of the security implications of remote computer control

API Key Authentication

The service supports optional API key authentication for securing service-to-service communication:

  1. Enable Authentication: Set API_KEY_ENABLED=true in your .env file

  2. Set API Key: Configure API_KEY=your-secret-api-key-here in your .env file

  3. Pass API Key in Requests: Include the API key in request headers:

    • X-API-Key: your-secret-api-key-here (recommended)

    • Authorization: Bearer your-secret-api-key-here (alternative)

Excluded Paths (no authentication required):

  • /health - Health check endpoint

  • /docs - API documentation

  • /openapi.json - OpenAPI schema

  • /redoc - Alternative API documentation

MCP Client Usage with API Key:

from fastmcp import Client from fastmcp.client.transports import StreamableHttpTransport # Create transport with API key transport = StreamableHttpTransport( url="http://localhost:8001/mcp", headers={"X-API-Key": "your-secret-api-key-here"} ) # Create client with transport client = Client(transport) async with client: response = await client.call_tool("move_mouse", {"x": 100, "y": 200})

Logging

The service uses Loguru for structured logging with the following features:

  • Request ID Tracking: Each request gets a unique ID that appears in all log entries

  • Environment-aware: Console output in development, file logging in production

  • Structured Format: Includes timestamp, level, request ID, module, function, and line number

Log files are stored in the logs/ directory:

  • app_YYYY-MM-DD.log: General application logs

  • error_YYYY-MM-DD.log: Error logs only

In development mode, logs are only output to the console. In production mode, logs are written to both console and files.

Testing

Run tests using pytest:

# Run all tests uv pytest # Run specific test file uvpytest tests/test_mcp_client.py # Run with verbose output pytest -v

The test suite includes:

  • test_local_mcp_client.py: Tests for local MCP server with HTTP transport (direct tools)

  • test_stdio_mcp_client.py: Tests for local MCP server with stdio transport (direct tools)

  • test_mcp_client.py: Tests for remote MCP server with client-based tools (requires endpoint parameter)

Troubleshooting

Port Already in Use

If you get a port already in use error:

# Change ports in .env file PORT=8002 MCP_PORT=8003

MCP Connection Issues

For HTTP transport, ensure the MCP server is running and accessible:

# Test MCP endpoint curl http://localhost:8001/mcp # Test with API key (if enabled) curl -H "X-API-Key: your-secret-api-key-here" http://localhost:8001/mcp

For stdio transport, ensure the MCP server is started with stdio mode:

# Start MCP server in stdio mode uv python mcp_local.py stdio

API Key Authentication Issues

If you're getting authentication errors:

  • Verify API_KEY_ENABLED is set correctly in .env

  • Check that API_KEY matches between client and server

  • Ensure the API key is passed in the X-API-Key header or Authorization: Bearer <key> header

  • Check that the request path is not in the excluded paths list

Screenshot Issues

If screenshot functionality fails:

  • Check Python version compatibility (requires Python >= 3.12)

  • Verify display permissions on macOS/Linux

  • Ensure PyAutoGUI and its dependencies are properly installed

License

MIT license

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

This server cannot be installed

-
security - not tested
A
license - permissive license
-
quality - not tested

How are these scores calculated?

Resources

New MCP Servers

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/stonehill-2345/mcp-autogui-multinode'

If you have feedback or need assistance with the MCP directory API, please join our Discord server