ue5-gameplay-mcp

An MCP server that plays a running Unreal Engine 5 game. It accepts virtual gamepad, keyboard, and mouse input and returns screen capture, log lines, and UMG status.

This is not an engine plugin; it is a client. Two plugins already handle the in-engine work, and each holds its own port. This server dials into both and exposes them as a single tool interface.

A small amount of C++ was added to both plugins specifically for this server. They remain independent and keep their own protocols. The additions are backward compatible, so even older clients can still communicate with a rebuilt game:

• FImageMeta.SourceSize — the game's back-buffer size. It is packed into a previously reserved area to keep the struct at 24 bytes, so the client can map a specific point on a downscaled capture back to actual window pixels.

• CMD_GET_GAME_STATE (520) and IRemoteGameStateProvider — more on this below.

Setup

cd ue5_gameplay_mcp
uv sync

MCP Python SDK v2 (mcp.server.MCPServer) is required.

Related MCP server: VERA MCP Server

How to run

Start the game first. In this project, a standalone game is run from the editor binary because the Game target exits immediately in a project that has not been cooked:

"C:/Program Files/Epic Games/UE_5.8/Engine/Binaries/Win64/UnrealEditor.exe" "<PATH>/MyProject.uproject" -game -windowed -resx=1280 -resy=720 -log -nosplash

Next, register the server. The .mcp.json in the project root already does this, so Claude Code will pick it up automatically. The manual command equivalent is:

claude mcp add ue5-gameplay -- uv run --directory <PATH>/ue5_gameplay_mcp -m ue5_gameplay_mcp

The server uses a lazy connection, so the actual startup order does not matter. If it starts before the game does, it will connect on the first tool call.

Options: --host, --console-port, --capture-port, --format, --quality, --max-size, --grid-step, --transport streamable-http --mcp-port 14102.

Tools

Session — game_connect, game_status, game_reset_input

Observation — game_observe, game_state, game_log, game_wait_for_log

Actions — game_pad, game_pad_sequence, game_key, game_mouse, game_console, game_time_scale

UMG — game_ui_dump, game_ui_click, game_ui_focus

API design and the reason for it

An agent round trip takes several seconds, but the game runs at 60 Hz. Since frame-by-frame actions are unrealistic, the design works as follows:

• All action tools take a duration and perform a press / hold / release sequence locally, paced to the communication speed. One round trip carries one intent, not one frame.

• Action tools observe by default. game_pad(ly=1.0, duration=0.5) moves forward and returns the resulting frame. This takes half as many round trips as doing the action and the observation separately.

• game_pad_sequence packs an entire combo into one call when the timing of the inputs matters more than checking between them.

• game_time_scale(0.2) buys in-game time when precision at a particular moment is required.

• hold=True keeps the input applied between turns so the character keeps moving while the agent is thinking. game_reset_input clears it.

Numbers, not pixels

game_state returns the level, world time, pause/time dilation, player pawn transform, velocity, movement mode, camera, and the distance and normalized screen position of the nearest actor. These are the same 0–1 coordinates game_mouse receives, so you can immediately aim at a target you find in the status report. The cost is a fraction of image processing, and it will not misread HUD numbers.

game_observe(state=True) folds this into what an observation, and `game_pad(..., state=True)' folds it into an action, so movement and verification still take only one round trip.

On a real map, most nearest actors are background objects, so the report also includes class_counts (a survey of all objects within the radius). Read it once, and you can narrow it down with class_filter="Enemy".

Adding game-specific numbers

The built-in report needs no game-side query. For anything only that project knows — health, score, quest flags, etc. — implement IRemoteGameStateProvider (Plugins/RemoteConsole2/Source/RemoteConsole2/RemoteGameState.h) on any actor and return the string of a JSON object:

FString AMyGameMode::GetRemoteGameState_Implementation()
{
    return FString::Printf( TEXT("{\"score\":%d,\"wave\":%d}"), Score, Wave );
}

Since it's a BlueprintNativeEvent, it can be overridden even in Blueprint-only projects. All returned values are stored under custom, keyed by actor name. Providers are collected regardless of the distance filter, so even a scorekeeper handled at the origin will send a report. Text that is not valid JSON is not discarded; it is passed through as its raw string, so even simple Printf debugging while starting up is helpful.

Three ways to drive the menu (recommended order)

1. game_ui_dump + game_ui_click — exact and fast, but only knows widgets registered through UMG. Games with a custom Slate UI get nothing back, and the tool explicitly says so instead of hanging.

2. Pad navigation — game_pad(buttons=["DOWN"]), game_pad(buttons=["A"]). Works with almost any game.

3. Look & click — game_observe(grid=True) overlays a labeled 0–1 coordinate grid on the capture. Read the target from the image and pass the same numbers to game_mouse(x=..., y=...). It is independent of user resolution and works no matter how the UI is built.

Conventions

• Sticks follow UE's specification: ly=+1 is forward. (The protocol flips the Y axis, but our code flips it back, so the tool API matches what the game's own axis mapping means.)

• Mouse coordinates are normalized to 0–1 with the top-left as origin, and are translated to pixels using the game's actual back-buffer size. Since the capture reports its resized dimensions, this source size is probed separately.

• game_observe returns only new log lines since the previous observation, so the same output is never resent, even during a long session.

Known issues and unfinished parts

• When hosting the game from the editor binary, console commands are routed through Python. FGameAppInterface::ExecConsoleCommand dispatches to the implementation of IConsoleCommandExecutor [0]; if the editor's Python plugin is loaded, that slot is Python instead of Cmd. So a plain stat fpscomes back as aSyntaxError. On first use the server probes this, and if it detects the problem, it wraps the command with unreal.SystemLibrary.execute_console_command. Packaged builds have no Python executor, so this solution is not needed. You can also override it with game_console(via="cmd")`.

• Screen clicks assume the capture fills the entire game window. This is true with -game -windowed. In a letterboxed fullscreen mode, the reported source size includes the black bars, so the mapping is off.

• game_state traverses every actor in the level on each call. At normal map scale this is fine, but a streaming open world would probably need a spatial query instead of TActorIterator.

Tests

uv run test/smoke_test.py

Communicates directly with the game and outputs smoke_*.jpg, allowing you to check the capture and grid overlays visually.

uv run test/mcp_client_test.py

Starts the server via standard input/output (stdio) as a real MCP client, and runs and tests all tools, including error paths.