Skip to main content
Glama

Universal Game MCP (UGM) - V2.0

Windows CI Python 3.12 MCP SDK v2 Version: 2.0.0 License: MIT Tested on Ryzen 5 5500U

English | Tiếng Việt (Vietnamese Guide) | HOI4 Guide | Skill Packs

Universal Game MCP (UGM) is a lightweight, CPU-friendly Model Context Protocol (MCP) server that empowers external AI agents (Claude, Gemini, GPT-4, etc.) to autonomously explore, navigate, learn, and play offline strategy and simulation games without requiring manual scripting, game-specific logic, secondary agent runtimes, or model fine-tuning.

The AI model provides high-level reasoning and decision-making. Universal Game MCP provides:

  • Eyes: DXGI/GDI screen capture, event-driven OCR (RapidOCR PP-OCRv4), icon detection, and adaptive UI element exploration.

  • Hands: DirectInput mouse and keyboard control with bounded coordinate safety and dry-run simulation.

  • Memory: Persistent SQLite knowledge store with WAL mode, perceptual hashing (dHash), point-in-time checkpoints, and rollback.

  • Navigation: Multi-signal UI state classification and Dijkstra shortest-path menu traversal.

  • Self-Learning & Exploration: Autonomous interactive UI probing, 4-state knowledge lifecycle (candidate → verified → suspect → rejected), and cross-game concept generalization.


⚡ Measured Benchmark & Performance Profile

Benchmarked on AMD Ryzen 5 5500U (6 Cores / 12 Threads, Windows 11 x64, integrated Radeon graphics) under pure CPU execution:

Subsystem

Measured Latency

Throughput / Notes

Synthetic In-Memory Fixture Latency

1.79 ms

In-memory memcpy baseline buffer grab

Visual Diff Verification

5.48 ms

Real-time state verification (cv2.absdiff + threshold)

ROI Template Match (OpenCV)

43.97 ms

13.2x faster than full screen (Multi-scale + NMS)

Full-Screen Template Match

579.94 ms

Full 1080p scan

Event-driven ROI OCR

831.89 ms

PP-OCRv4 CPU on target bounding box

Full-Screen 1080p OCR

2,185.15 ms

Heavy; avoided at runtime via ROI crop

RAM Footprint (Steady State)

136.12 MB

Entire engine, OCR runtime & SQLite in RAM (< 150 MB)

NOTE

Synthetic in-memory fixture benchmarks are explicitly distinguished from live DXGI desktop duplication captures to prevent misleading performance claims.


Related MCP server: local-game-mcp

🏗 Architecture (V2 Autonomous Exploration Infrastructure)

                    +-----------------------------------+
                    |        AI Client / LLM Agent      |
                    +-----------------+-----------------+
                                      | (MCP SDK v2 JSON-RPC)
                                      v
                    +-----------------------------------+
                    |    Universal Game MCP Server      |
                    |          (ugm.server)             |
                    +-----------------+-----------------+
                                      |
         +----------------------------+----------------------------+
         |                            |                            |
         v                            v                            v
+------------------+         +------------------+         +------------------+
|    ugm.core      |         |   ugm.memory     |         |  ugm.navigation  |
| - capture (live) |         | - database.py    |         | - recognizer.py  |
| - input (direct) |         |   (SQLite WAL)   |         |   (dHash + OCR)  |
| - ocr (rapidocr) |         | - storage.py     |         | - graph.py       |
| - icon (opencv)  |         | - checkpoints    |         |   (Dijkstra)     |
| - explorer (v2)  |         | - cross_game     |         | - navigator.py   |
| - safety & stop  |         | - dHash & pruner |         +------------------+
+------------------+         +------------------+                  |
         |                            |                            |
         +----------------------------+----------------------------+
                                      |
                                      v
                             +------------------+
                             |    ugm.skills    |
                             | - engine.py      |
                             | - learner.py     |
                             | - loader.py      |
                             +------------------+

Core Subsystems

  1. Autonomous Exploration (ugm.core.explorer): Generic element discovery, interactive contour and text bounding box grouping, non-destructive probing actions, and layout zone segmentation.

  2. Persistent Knowledge & Lifecycle (ugm.memory.database, ugm.memory.storage): SQLite with WAL mode, point-in-time snapshot checkpoints, versioning, rollback, perceptual hash indexing (dHash), and 4-state knowledge promotion.

  3. Multi-Signal UI State Recognizer (ugm.navigation.recognizer): Adaptive layout classifier combining perceptual dHash, prominent text anchors, panel geometry, and confidence metrics (known, ambiguous, unknown).

  4. Navigation Graph (ugm.navigation.graph, ugm.navigation.navigator): Dynamic directed graph where nodes are UI states and edges are verified executable actions. Plans lowest-cost routes via Dijkstra with automatic replanning on mismatch.

  5. Verified Self-Learning Engine (ugm.skills.learner): Procedural learning pipeline: Observe → Identify → Plan → Act → Verify → Learn → Reuse. Curated packs remain read-only; learned skills are incrementally validated and promoted.

  6. Skill Execution Engine (ugm.skills.engine): Sandboxed runtime with allowlisted action primitives (press_keys, click_target, locate_and_click, click_selector, hover, wait, verify_state), precondition/postcondition checks, and emergency killswitch.

  7. Cross-Game Concept Store (ugm.memory.storage): Registers and queries abstract strategy patterns (e.g., economy loops, pause toggles, production queues) shared across multiple games.


🛠 Complete MCP Tool Reference (34 Tools)

1. Perception, Input & Safety Tools (12 Tools)

Tool Name

Parameters

Description

get_game_window

window_title, process_name

Finds and attaches to target game window; returns geometry, DPI scale, and client rect.

capture_screen

roi, save_to_disk, include_base64

Captures current game frame with client rect coordinates.

read_text

roi, enhance_small_text

Event-driven RapidOCR reading on normalized ROI.

locate_ui_element

selector_id, icon_name, text, threshold, roi

Multimodal target resolution with ambiguity detection and escalation.

locate_icon

icon_name, threshold, roi

Multi-scale OpenCV template matching with NMS.

hover_target

x, y, duration_ms

Hovers mouse at normalized (x, y) and reads popup tooltip.

click_target

x, y, selector_id, dry_run, verify

Clicks target dynamically; verifies visual state before confirming.

press_keys

keys, duration_ms, dry_run

Sends DirectInput keystrokes (e.g. ['space'] for pause).

verify_state

expected_text, roi, pre_capture_id

Computes visual difference and validates state transition.

verify_selection

expected_text, roi, pre_capture_id

Verifies active UI element selection and text status.

emergency_stop

reason

Instant global emergency killswitch halting all input dispatch.

save_training_sample

tag, label, metadata

Opt-in recording of annotated game frames to data/samples/.

2. Memory, Navigation & Skill Execution Tools (11 Tools)

Tool Name

Parameters

Description

list_game_packs

none

Lists discovered curated packs and registered games in persistent memory.

get_ui_state

game_id, hint_state

Classifies current game UI state (e.g. main_map, production) with confidence and evidence.

memory_search

game_id, query, search_type, limit

Searches persistent SQLite memory across states, transitions, skills, or action history.

get_navigation_plan

target_state, start_state, game_id

Calculates optimal multi-step navigation route using Dijkstra algorithm.

navigate_to

target_state, game_id, max_retries, dry_run

Safely traverses menus step-by-step with state verification and replanning.

execute_skill

skill_id, game_id, params, dry_run

Executes a curated or learned Game Skill with precondition/postcondition checks.

learn_skill

skill_id, name, steps, game_id, preconditions, post_verification

Saves newly discovered action sequence as candidate skill in persistent memory.

list_learned_skills

game_id, status

Lists procedurally learned skills, validation track records, and status.

get_learning_status

game_id

Reports persistent memory statistics (states, transitions, action history, skills).

delete_learned_memory

game_id, skill_id, scope

Safely deletes scoped memory or individual learned skill.

index_game_assets

game_path, game_id

Read-only extraction of UI metadata and sprites from installed game files into local cache.

3. V2 Autonomous Exploration & Knowledge Lifecycle Tools (11 Tools)

Tool Name

Parameters

Description

discover_and_attach_game

window_title, process_name, game_id

Auto-detects active foreground/named game, registers baseline resolution, and initializes empty knowledge store.

explore_interactive_elements

roi, min_confidence, max_elements

Scans active screen/ROI for interactive elements (buttons, icons, tabs, inputs) with bounding boxes.

probe_action

action, x, y, keys, roi, duration_ms, dry_run

Non-destructive probing action with pre/post capture diff analysis and isolated change tracking.

manage_knowledge_state

item_type, item_id, new_status, reason, game_id

Updates lifecycle state (candidate, verified, suspect, rejected) of states, transitions, or skills.

create_knowledge_checkpoint

label, game_id

Creates an atomic point-in-time snapshot checkpoint of game knowledge.

rollback_knowledge_checkpoint

checkpoint_id, game_id

Restores knowledge state to a previous point-in-time checkpoint.

list_knowledge_checkpoints

game_id

Lists all available checkpoints with creation timestamps and metadata.

query_cross_game_concepts

category, tag

Queries reusable abstract concepts and gameplay patterns applicable across titles.

register_cross_game_concept

concept_id, name, category, abstract_pattern, tags

Registers an abstract gameplay pattern into cross-game knowledge store.

optimize_skill

skill_id, game_id, strip_redundant, tighten_delays

Optimizes execution latency of a verified skill by removing redundant delays and probes.

prune_temporary_data

game_id, older_than_days, dry_run

Safely cleans unpromoted candidates, rejected items, and transient exploration data.


🎮 Game Skill Packs (Curated & Learned)

game_skills/
├── README.md                      # Skill pack architecture & JSON schemas
├── _template/                     # Starter template for new games
└── hoi4/                          # Hearts of Iron IV Reference Pack
    ├── manifest.json              # Process names, resolutions, dpi-awareness
    ├── skills/                    # Curated skills (pause_unpause, open_production, increase_speed, select_army_group)
    ├── selectors/                 # Visual and OCR selectors (topbar_factories, political_power, pause_button)
    └── templates/                 # Reference PNG icons for template matching
  • Curated Skill Protection: Curated skills in game_skills/ are read-only at runtime and can never be overwritten by self-learning calls.

  • Procedural Promotion: Learned skills start as candidate and promote to verified after ≥ 3 successful validations with confidence ≥ 0.85.

  • Security Sandboxing: Execution of arbitrary shell scripts or code is blocked; only allowlisted primitives are permitted.


🚀 Quickstart

1. Installation

Requires Python 3.12 on Windows 10/11 x64:

# Clone repository
git clone https://github.com/QuafdaPoet/universal-game-mcp.git
cd universal-game-mcp

# Create Python 3.12 virtualenv
uv venv --python 3.12 .venv
# or: python -m venv .venv

# Activate virtualenv
.\.venv\Scripts\activate

# Install dependencies in editable mode
pip install -e .

2. Generate Test Fixtures & Run Offline Tests

# Generate offline test fixtures
python scripts/generate_fixtures.py

# Run automated tests (100% offline, no game required)
pytest -v

3. Run CPU Latency & RAM Profiling Benchmark

python scripts/benchmark_cpu_ram.py

4. Starting the Universal Game MCP Server

# Run with stdio transport (Claude Desktop, Cursor, or Antigravity IDE)
python -m ugm.server

# Or enforce dry-run server-wide (inviolable hardware simulation)
python -m ugm.server --dry-run

MCP Client Configuration (claude_desktop_config.json):

{
  "mcpServers": {
    "universal-game-mcp": {
      "command": "C:\\Users\\<USER>\\...\\universal-game-mcp\\.venv\\Scripts\\python.exe",
      "args": ["-m", "ugm.server"]
    }
  }
}

🔒 Safety and Anti-Cheat Policies

  • Offline & Single-Player Only: Exclusively intended for single-player, offline strategy/simulation games.

  • Strict Anti-Cheat Blocker: Active process checks immediately abort and trigger emergency stop if anti-cheat services (EAC, BattlEye, Vanguard, Ricochet, etc.) are detected.

  • Server-Wide Dry-Run: When launched with --dry-run, tool calls cannot override it with dry_run=False.

  • Emergency Stop Killswitch: The emergency_stop tool instantly locks out mouse and keyboard inputs.


📄 License & Attribution

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • F
      license
      Not graded
      quality
      D
      maintenance
      Enables AI agents to play local Windows games through low-level keyboard/mouse input, screen capture, OCR, and per-game profiles for semantic actions.
      -
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables AI assistants to autonomously control Windows 11 and 10 desktops via sub-10ms screen capture, native UI Automation element inspection, and zero-lag keyboard and mouse input. Combines a visual plane with a semantic plane and stall detection so agents can operate real applications reliably without vision-only guessing.
      262 npm
      171 PyPI
      7
      MIT
    • A
      license
      A
      quality
      C
      maintenance
      Enables AI assistants to interact with running games in real time — introspecting game state, simulating keyboard and mouse input to actually play, capturing screenshots, executing Lua code, hot-reloading files, and receiving push notifications when state changes.
      8
      MIT