AI Gaming Agent MCP Server
Integrates with Ollama to provide local vision-language model analysis of screen content, enabling fast visual processing without relying on cloud APIs.
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
AI Gaming Agent MCP Server
MCP (Model Context Protocol) server that enables AI agents like Claude to remotely control gaming PCs for automated gameplay.
Master Claude (Orchestrator)
|
| MCP Protocol (JSON-RPC over HTTP/SSE)
|
+---> PC #1 (MCP Server :8765) --> PyAutoGUI + Optional VLM
+---> PC #2 (MCP Server :8765) --> PyAutoGUI + Optional VLM
+---> PC #N (MCP Server :8765) --> PyAutoGUI + Optional VLMFeatures
23 MCP Tools: Screen capture, mouse/keyboard control, file operations, system commands, workflow automation
Workflow Automation: Chain multiple actions into single commands with
run_workflowanddemo_terminal_workflowMulti-Monitor Support: Target specific monitors for screenshots and actions
Dual Transport Modes: HTTP/SSE for remote control, stdio for local clients
Optional Local VLM: Use Ollama (Qwen2.5-VL, Moondream) for fast local screen analysis
Security-First: Bearer token auth, Host/Origin allow-lists, path restrictions, command blocklist
Cross-Platform: Windows, Linux, macOS with auto-detection
Related MCP server: local-game-mcp
Quick Start
Prerequisites
Python >=3.12 and Git. Ubuntu 24.04 supplies Python 3.12; older distributions may need a newer interpreter and matching venv/Tk development packages.
For automation, a logged-in graphical session: on Linux, X11 or a compatible session with
DISPLAYset and access to the target desktop. Installation andgaming-agent --helpwork headlessly, withoutDISPLAY.
On Debian/Ubuntu:
sudo apt update
sudo apt install -y git ca-certificates python3 python3-venv python3-tk python3-dev build-essential scrot xclip wmctrlIn a root container, omit sudo (it may not be installed). Run desktop automation
as the logged-in desktop user, not root.
Fedora equivalents: python3 and python3-pip (venv support), python3-tkinter
for Tk, and python3-devel/gcc for native builds. Arch uses python,
python-pip, tk, and base-devel. Both also need git, ca-certificates,
scrot, xclip, and wmctrl. On macOS/Windows, install Python >=3.12 with Tk and Git.
Installation
Install directly from Git, targeting the application subdirectory. In a Linux/macOS shell, create and activate a virtual environment first:
python3 -m venv ~/.gaming-agent/venv
source ~/.gaming-agent/venv/bin/activate
pip install "git+https://github.com/developerz-ai/ai-gaming-agent-mcp.git#subdirectory=apps/ai-gaming-agent-mcp"
gaming-agent --helpOn Windows, create the environment with your Python 3.12+ interpreter and activate
%USERPROFILE%\.gaming-agent\venv\Scripts\activate.bat in cmd.exe before installing.
Alternatively, if uv is installed,
run in an isolated environment with uvx. This does not persistently install
gaming-agent into your shell; Git and the system prerequisites above still apply:
uvx --from "git+https://github.com/developerz-ai/ai-gaming-agent-mcp.git#subdirectory=apps/ai-gaming-agent-mcp" gaming-agent --helpFor development, use an editable checkout instead (with the virtual environment active):
git clone https://github.com/developerz-ai/ai-gaming-agent-mcp.git
cd ai-gaming-agent-mcp/apps/ai-gaming-agent-mcp
pip install -e .Start the Server
The server supports two transport modes:
HTTP/SSE Transport (Recommended for Remote Control):
gaming-agent serve --transport http --port 8765 --password your-secret-passwordStdio Transport (For Local MCP Clients):
gaming-agent serve --transport stdioConnect from Claude Code
The HTTP server speaks the legacy HTTP+SSE MCP transport (GET /mcp opens the
stream, messages are POSTed to /mcp/messages) over plain HTTP with a Bearer
token. Claude Code connects to it directly. --transport must come before the
server name:
claude mcp add --transport sse gaming-pc http://YOUR-PC-IP:8765/mcp --header "Authorization: Bearer your-secret-password"Check the connection with claude mcp list. Add --scope user to make the
server available in every project. Claude Code flags the SSE transport as
deprecated but still supports it.
Connect from Claude Desktop
For HTTP/SSE Transport:
Claude Desktop's claude_desktop_config.json only accepts stdio servers (a
command with args). An entry with "transport": "sse" and a url is not
valid there. Reach the HTTP/SSE server through the
mcp-remote stdio bridge, which needs
Node.js (for npx) on the machine running Claude Desktop.
The config file is at:
macOS:
~/Library/Application Support/Claude/claude_desktop_config.jsonWindows:
%APPDATA%\Claude\claude_desktop_config.json
This README does not cover Claude Desktop's separate Linux beta. Use Claude Code above on Linux.
{
"mcpServers": {
"gaming-pc": {
"command": "npx",
"args": [
"-y",
"mcp-remote",
"http://YOUR-PC-IP:8765/mcp",
"--transport",
"sse-only",
"--allow-http",
"--header",
"Authorization:${AUTH_HEADER}"
],
"env": {
"AUTH_HEADER": "Bearer your-secret-password"
}
}
}
}--transport sse-onlyforces the legacy SSE transport this server speaks.--allow-httpis required for a plainhttp://URL; use it only on a trusted private network.The token goes in the
AUTH_HEADERenv var and is referenced asAuthorization:${AUTH_HEADER}(no space after the colon). This avoids a known problem with spaces inargson Windows.For several PCs, add one entry per PC with a distinct name and host.
Custom connectors do not fit. Claude Desktop's Settings > Connectors > Add
custom connector needs a public HTTPS URL and OAuth (or no auth). A LAN
http:// URL with a static Bearer header cannot be added there, which is why
the bridge is used.
Client configuration validation
The documented configurations were exercised against a local server at
http://127.0.0.1:8765/mcp with its configured Bearer token:
Claude Code 2.1.306:
claude mcp add --transport sse ...reportedStatus: ✔ Connectedfromclaude mcp get.Claude Desktop bridge (
mcp-remote0.16.6): the documentednpxcommand with--transport sse-only,--allow-http, andAuthorization:${AUTH_HEADER}connected withSSEClientTransportand reportedProxy established successfully between local STDIO and remote SSEClientTransport.
The second check exercises the exact stdio bridge command that Claude Desktop
launches from claude_desktop_config.json; it does not require a signed-in
desktop application session.
The MCP endpoints validate the Host and Origin headers against
server.allowed_hosts / server.allowed_origins, which default to loopback
only. A remote PC must list its own address (e.g. 192.168.1.100:8765) in
server.allowed_hosts, otherwise the request is refused with 421.
For Stdio Transport:
{
"mcpServers": {
"gaming-pc": {
"command": "gaming-agent",
"args": ["serve", "--transport", "stdio"]
}
}
}Quick Test
Test the complete automation capability with a single command:
# Start the server
gaming-agent serve --transport http --password test123
# In Claude Desktop (after connecting), ask:
"Use the demo_terminal_workflow tool to open a terminal, type 'echo hello world', and close it"This will:
✓ Auto-detect your terminal (gnome-terminal, konsole, xterm, Terminal.app, cmd)
✓ Open a new terminal window
✓ Type "echo hello world"
✓ Press Enter to execute
✓ Capture a screenshot for verification
✓ Close the terminal with the appropriate hotkey
Available Tools
Total: 23 MCP Tools
Workflow Tools (New!)
Tool | Description |
| Execute a sequence of tool actions with optional delays |
| Complete demo: open terminal, type command, execute, screenshot, close |
Screen Tools
Tool | Description |
| Capture current screen (returns base64 PNG) |
| Get screen dimensions |
| Use local VLM to analyze screen content (requires Ollama) |
Mouse Tools
Tool | Description |
| Click at coordinates |
| Double-click at coordinates |
| Move mouse cursor |
| Drag from current position |
| Scroll mouse wheel |
| Get current cursor location |
Keyboard Tools
Tool | Description |
| Type a string of text (supports fast paste mode via clipboard) |
| Press a single key |
| Press key combination |
Fast Text Input with Paste
The type_text tool supports fast clipboard-based paste mode for significantly faster text input:
{
"tool": "type_text",
"args": {
"text": "long command or text here",
"use_paste": true
}
}Benefits:
10x faster than character-by-character typing for long text
Ideal for: Pasting long commands, scripts, credentials
How it works: Copies text to clipboard, then uses Ctrl+V (Linux/Windows) or Cmd+V (macOS) to paste
Default:
use_paste=false(uses character-by-character typing)
When to use:
✓ Large blocks of text
✓ Complex commands with special characters
✓ When speed matters more than realtime character visibility
✗ Games that don't support paste input
✗ When character-by-character input is explicitly required
File Tools
Tool | Description |
| Read file contents |
| Write content to file |
| List directory contents |
| Upload file to PC |
| Download file from PC |
System Tools
Tool | Description |
| Run shell command |
| Get CPU/RAM/GPU usage |
| List open windows |
| Bring window to foreground |
Workflow Automation
run_workflow - Composite Command Execution
Execute multiple tools in sequence with a single command. Perfect for complex automation tasks.
Example: Open terminal and run command
{
"steps": [
{
"tool": "execute_command",
"args": {"command": "gnome-terminal"},
"wait_ms": 1500,
"description": "Open terminal"
},
{
"tool": "type_text",
"args": {"text": "ls -la"},
"wait_ms": 200,
"description": "Type command"
},
{
"tool": "press_key",
"args": {"key": "enter"},
"wait_ms": 1000,
"description": "Execute command"
},
{
"tool": "screenshot",
"args": {},
"description": "Capture result"
}
]
}Step Fields:
tool(required): Name of the tool to executeargs(optional): Arguments to pass to the toolwait_ms(optional): Milliseconds to wait after this stepdescription(optional): Human-readable step descriptioncontinue_on_error(optional): Continue workflow if this step fails
Returns:
{
"success": true,
"total_steps": 4,
"completed_steps": 4,
"failed_step": null,
"results": [...],
"total_time_ms": 3523,
"error": null
}demo_terminal_workflow - Ready-Made Terminal Demo
A convenience tool that demonstrates the full automation capability in one call.
Usage:
{
"text": "echo hello world",
"terminal_wait_ms": 2000,
"post_type_wait_ms": 500,
"post_enter_wait_ms": 1000,
"capture_screenshot": true,
"close_terminal": true
}What it does:
Auto-detects platform terminal (gnome-terminal, konsole, xterm, Terminal.app, cmd)
Opens the terminal application
Waits for terminal to fully load
Types the provided command
Presses Enter to execute
Waits for command output
Captures screenshot for verification (optional)
Closes terminal with platform-appropriate hotkey (optional)
Returns:
{
"success": true,
"terminal_command": "gnome-terminal",
"platform": "Linux",
"text_typed": "echo hello world",
"screenshot": {"success": true, "image": "...", ...},
"steps_completed": ["detect_terminal", "open_terminal", "wait_for_terminal",
"type_text", "press_enter", "capture_screenshot",
"close_terminal"],
"total_time_ms": 4523,
"error": null
}Platform Support:
Linux: gnome-terminal, konsole, xfce4-terminal, mate-terminal, tilix, terminator, xterm
macOS: Terminal.app
Windows: cmd.exe
Configuration
Create ~/.gaming-agent/config.json:
{
"server": {
"host": "0.0.0.0",
"port": 8765,
"password": "your-secure-password",
"allowed_hosts": ["localhost:*", "127.0.0.1:*", "[::1]:*"],
"allowed_origins": ["http://localhost:*", "http://127.0.0.1:*", "http://[::1]:*"]
},
"vlm": {
"enabled": false,
"provider": "ollama",
"model": "qwen2.5-vl:3b",
"endpoint": "http://localhost:11434"
},
"security": {
"allowed_paths": ["/home/user/games", "C:\\Games"],
"blocked_commands": ["rm -rf", "format", "del /f"],
"max_command_timeout": 30
}
}Enabling VLM (Optional)
To use the analyze_screen tool with local vision models:
Install Ollama (if not already installed):
curl -fsSL https://ollama.com/install.sh | shPull a vision model:
ollama pull qwen2.5-vl:3b # Lightweight, fast # or ollama pull moondream # AlternativeInstall the Ollama Python client in the activated virtual environment. The
vlmextra does not install the Ollama server from step 1:pip install "ai-gaming-agent[vlm] @ git+https://github.com/developerz-ai/ai-gaming-agent-mcp.git#subdirectory=apps/ai-gaming-agent-mcp"Enable in config (
~/.gaming-agent/config.json):{ "vlm": { "enabled": true, "provider": "ollama", "model": "qwen2.5-vl:3b", "endpoint": "http://localhost:11434" } }Use in workflows:
{ "tool": "analyze_screen", "args": { "prompt": "What is the current health percentage?" } }
Deployment Options
Option A: Claude Does All Vision (Simplest)
Claude analyzes all screenshots
Gaming PCs are simple executors
No GPU needed on gaming PCs
Use HTTP/SSE transport with Bearer auth
Option B: Hybrid with Local VLM (Recommended)
Claude for high-level decisions and orchestration
Local VLM (Qwen2.5-VL, Moondream) for fast visual processing
Best balance of speed and intelligence
Reduces API costs and latency
Option C: Full Local (Privacy)
Local orchestrator (e.g., Qwen3-72B via Ollama)
No cloud APIs, complete privacy
Requires powerful hardware (GPU recommended)
Use stdio transport for local control
Practical Examples
Example 1: Terminal Automation
# Ask Claude: "Run the demo_terminal_workflow with the command 'uname -a'"
# Result: Opens terminal, runs command, captures output, closesExample 2: Multi-Step Workflow
# Ask Claude: "Create a workflow that:
# 1. Opens a file browser
# 2. Navigates to Downloads
# 3. Takes a screenshot
# 4. Closes the window"
# Claude will use run_workflow with execute_command, type_text, screenshot, hotkeyExample 3: Game Automation with VLM
# Ask Claude: "Use analyze_screen to check if the game menu is visible,
# then click the 'Start Game' button at coordinates you detect"
# Claude will:
# 1. Call analyze_screen with prompt "Is there a Start Game button? Where?"
# 2. Use VLM response to determine coordinates
# 3. Call click tool with detected coordinatesExample 4: Batch File Operations
# Ask Claude: "Create a workflow that backs up all .save files from
# C:\Games\MyGame to C:\Backups\saves-{date}"
# Claude will use run_workflow with list_files, read_file, write_fileSecurity
Always use strong, unique passwords (min 16 chars, random)
The MCP endpoints (
/mcp,/mcp/messages) validate theHostandOriginheaders againstserver.allowed_hosts/server.allowed_origins, which default to loopback only. AHostoutside the allow-list is refused with421, a disallowedOriginwith403A remote Claude Desktop connection needs the PC's own host in
server.allowed_hosts(e.g.192.168.1.100:8765); without it every/mcprequest is refused with421Widen the allow-lists with
GAMING_AGENT_SERVER__ALLOWED_HOSTS/GAMING_AGENT_SERVER__ALLOWED_ORIGINS(JSON array values); these fill in the allow-lists only while~/.gaming-agent/config.jsondoes not set them - once the file carriesserver.allowed_hosts/server.allowed_originsthose entries are used as-is, so edit them there insteadLimit file access to game directories only in
allowed_pathsUse a VPN if accessing over internet (never expose to public)
The server speaks plain HTTP - terminate TLS/HTTPS at a reverse proxy such as nginx for production
The server has no built-in rate limiting, lockout or audit trail: put an external rate limiter in front of it and keep it on a private network
Regularly rotate passwords (weekly for high-security environments)
The server writes no log or audit files - it logs to stdout/stderr, so capture that output (
gaming-agent serve 2>&1 | tee -a server.log) and review it for failed authentication attemptsSet appropriate
max_command_timeoutto prevent runaway processes
Development
# Install with uv (recommended) - run from the repository root
(cd apps/ai-gaming-agent-mcp && uv sync --extra dev)
# Or with pip
(cd apps/ai-gaming-agent-mcp && pip install -e ".[dev]")
# Lint
(cd apps/ai-gaming-agent-mcp && uv run ruff check src tests)
# Run unit tests
(cd apps/ai-gaming-agent-mcp && uv run pytest tests/ --ignore=tests/integration -v)Testing
Unit Tests (CI)
Unit tests run in CI on every push/PR. They test configuration, file operations, and tool interfaces without requiring a display.
# Run unit tests only
(cd apps/ai-gaming-agent-mcp && uv run pytest tests/ --ignore=tests/integration -v)Integration Tests (Local Only)
Integration tests perform real GUI automation and require:
A real display (X11, Wayland, Windows, macOS)
tesseract-ocrfor OCR verificationpyautogui working with your display
These tests CANNOT run in CI because they need a real desktop environment.
# Install integration test dependencies
(cd apps/ai-gaming-agent-mcp && uv sync --extra integration)
# Install tesseract (Linux)
sudo apt install tesseract-ocr
# Install tesseract (macOS)
brew install tesseract
# Run integration tests locally
(cd apps/ai-gaming-agent-mcp && uv run pytest tests/integration -v)What Integration Tests Do
Test | Description |
| Captures real screen content |
| Uses OCR to read text from screen |
| Moves mouse cursor to position |
| Performs real mouse click |
| Types actual text |
| Opens terminal, types command, closes |
| Opens terminal, runs command, verifies output with OCR |
| Runs 7 GUI operations in sequence |
Terminal Workflow Test
The most comprehensive test opens a terminal, types a command, and verifies the output:
# What the test does:
1. Opens system terminal (gnome-terminal, konsole, xterm, etc.)
2. Types: echo "AGENT_TEST_abc12345"
3. Presses Enter
4. Takes screenshot
5. Runs OCR on screenshot
6. Verifies "AGENT_TEST_abc12345" appears in OCR output
7. Closes terminal with Alt+F4CI Configuration
CI runs on GitHub Actions with Python 3.12 only:
# .github/workflows/ci.yml
- Checkout code
- Install uv
- Install Python 3.12
- Install system deps (xvfb, scrot, python3-tk)
- Run ruff lint
- Run unit tests (integration tests excluded)Integration tests are auto-skipped in CI via the CI=true environment variable.
License
MIT
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
This server cannot be deployed
Maintenance
Related MCP Connectors
Connect AI assistants to your approved computer for files, commands, browser and desktop tasks.
Give your AI hands on every computer you own. Remote control of your Windows, Mac and Linux machines from one ChatGPT, Claude or Cursor chat — even from your phone. Ask it to find out why the office server is crawling, update every PC in the house, sort out your mom's laptop from another city, or ship the site from your work machine while you're on a train. It runs commands, reads files and takes a screenshot when it needs to see the screen. You decide what each computer allows, and you can cut any of them off in a second.
Real desktops for AI agents, on machines you choose: screenshot, type, click, hire more agents.
- DNS-DATA RemoteDeprecatedcom.dns-data
Connect AI assistants to authorized Windows and macOS devices for files, commands, and management.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceEnables AI agents to see and control your desktop with tools for screenshots, clicks, typing, and more, all locally on macOS and Windows.107 npmMIT
- FlicenseNot gradedqualityDmaintenanceEnables AI agents to play local Windows games through low-level keyboard/mouse input, screen capture, OCR, and per-game profiles for semantic actions.-
- AlicenseNot gradedqualityBmaintenanceEnables AI agents to control remote desktops through screen capture, mouse movement, and keyboard input.MIT
- AlicenseNot gradedqualityBmaintenanceEnables AI assistants to autonomously control Windows 11 and 10 desktops via sub-10ms screen capture, native UI Automation element inspection, and zero-lag keyboard and mouse input. Combines a visual plane with a semantic plane and stall detection so agents can operate real applications reliably without vision-only guessing.262 npm7MIT