Baba Is Agent

A local Model Context Protocol server that lets an AI agent play the Linux Steam version of Baba Is You using the real engine's state and rules.

No screenshots. No OCR. No reimplemented game simulation. A small Lua mod reports exact objects, coordinates, conditional rules, menus, and victory events. Gameplay actions call the engine's own movement and undo functions. An MCP-capable agent supplies the reasoning; this project does not include a universal puzzle solver.

Requirements

• Linux, Python 3.13+, and uv.

• A legally installed native Linux Steam copy of Baba Is You, app 736260.

• A game version with the global Data/Lua/ mod loader and the documented MF_* Lua API.

• For menus: X11/XWayland, libX11, and permission to write /dev/uinput. Gameplay stepping and telemetry do not need a virtual keyboard.

Tested locally on GNOME Wayland with the game running through XWayland. Native Wayland-only windows, Windows/Proton builds, and macOS are not currently supported. Other game builds, custom mods, and all possible puzzle mechanics have not been exhaustively tested.

Related MCP server: LÖVE2D MCP Server

Quick start

From a local clone:

uv sync --locked
uv run baba-mcp install

Restart the game after installing or upgrading the bridge. Then:

uv run baba-mcp doctor --live
uv run python examples/client.py get_board

The installer adds only our mod at Data/Lua/zz_baba_mcp.lua and creates a private transport directory at Data/Languages/baba_mcp/. It refuses to overwrite an unrecognized mod. It does not patch the executable, replace original Lua scripts, change unlocks, or edit save files. Ordinary gameplay still writes Steam saves. Back up your saves before automated play.

If Steam uses a different library:

export BABA_GAME_DIR="/path/to/steamapps/common/Baba Is You"
uv run baba-mcp install

You can also use baba-mcp --game-dir "/path/to/game" install.

Menu input permissions

Check uv run baba-mcp doctor first. If the device exists but is not writable, an administrator can grant a temporary ACL to your desktop user:

sudo setfacl -m "u:$(id -un):rw" /dev/uinput

If /dev/uinput is absent, the administrator may need to load the uinput kernel module. ACLs may need to be reapplied after reboot. Revocation:

sudo setfacl -x "u:$(id -un)" /dev/uinput

Do not run the MCP server as root or make /dev/uinput world-writable. This permission allows virtual input to the desktop, not just this game. The tool requests game focus, but focus can still be changed by the user/compositor. Do not type, switch windows, or let another agent control the desktop while menu input is being sent.

press assumes the normal game bindings: arrow keys, Enter, Escape, Space, Z, and R. Custom bindings may need to be restored in the game. Exact act operations do not depend on bindings.

Connect an MCP client

Use the configuration in examples/mcp.json, replacing the absolute project path. The server uses stdio, with diagnostics on stderr. Running baba-mcp without a subcommand is equivalent to baba-mcp serve.

Complete Quick start first: client registration does not install the game-side bridge. Run the commands below from the clone's root directory, with uv on PATH. "$PWD" is saved as an absolute path, so keep the clone at that location.

Start the client as your normal desktop user on the same machine as the game. The server needs the current desktop session's DISPLAY and, where used, XAUTHORITY for menu input. If you set BABA_GAME_DIR, make sure the client passes it to the server as well. Environment interpolation differs between clients; examples/client.py explicitly inherits the environment. Connect any of these clients, but do not let them control the game concurrently.

Pi

Pi supports environment interpolation in its MCP configuration. Register from your clone:

pi mcp add baba \
  --cwd "$PWD" \
  --env 'DISPLAY=${DISPLAY}' \
  --env 'XAUTHORITY=${XAUTHORITY}' \
  --exposure direct \
  -- uv run --project "$PWD" baba-mcp
pi mcp list

Run /reload in an existing Pi session after changing external configuration. To use a project-local registration instead, add --local and grant project trust when prompted. User configuration lives in ~/.pi/agent/mcp.json; project configuration in .pi/mcp.json. --exposure direct makes the game tools available without a tool-search step.

See the Pi MCP documentation. These commands require a Pi version with built-in pi mcp support.

Codex CLI

Register the local stdio server:

codex mcp add baba \
  --env "DISPLAY=${DISPLAY}" \
  --env "XAUTHORITY=${XAUTHORITY:-}" \
  -- uv run --project "$PWD" baba-mcp
codex mcp list

Restart an existing Codex session after changing configuration; use /mcp inside Codex to inspect the connection. The CLI saves the entry in ~/.codex/config.toml.

The command above saves the current values of the desktop variables. To forward fresh values whenever Codex starts, remove those two entries from [mcp_servers.baba.env] and add this field to the existing [mcp_servers.baba] table instead:

env_vars = ["DISPLAY", "XAUTHORITY"]

For project-only configuration, put the server's TOML tables in .codex/config.toml instead; Codex reads project configuration only in trusted projects. Add BABA_GAME_DIR to env_vars if needed. See the Codex MCP documentation.

Claude Code

Register for your user account (available across projects):

claude mcp add --transport stdio --scope user \
  --env "DISPLAY=${DISPLAY}" \
  --env "XAUTHORITY=${XAUTHORITY:-}" \
  baba -- uv run --project "$PWD" baba-mcp
claude mcp get baba

Restart an existing Claude Code session, then use /mcp to inspect the connection and approve tool use when prompted. User-scoped servers are stored in ~/.claude.json.

The command saves the current values of the desktop variables; update them if your desktop session changes. For a shared project registration, replace --scope user with --scope project: this writes .mcp.json, and Claude Code asks for approval before using its servers. The saved clone path and desktop values are machine-specific; review them before sharing that file. For a private registration limited to the current project, use --scope local instead. See the Claude Code MCP documentation.

Verify the connection

With the game running and the bridge installed, ask the agent to call baba.get_board (the client may display a prefix such as mcp__baba__get_board). It should return the current board and rules. A successful MCP connection alone does not prove the game bridge or menu input permissions work; use uv run baba-mcp doctor --live to diagnose those separately.

Tools

Example calls:

uv run python examples/client.py get_board
uv run python examples/client.py act '{"actions": "R8 U2 Z"}'
uv run python examples/client.py choose_menu '{"name": "resume"}'

Efficient agent use

Read get_board once, then send a complete planned route in one act call. Compact notation: U/D/L/R move, W waits, Z undoes; optional counts repeat an action (R8, U2). Each batch advances through native game frames, checks stop conditions after each move, and produces one final snapshot rather than reading/serializing the whole board after every move. Older installed bridges remain supported with a slower compatibility path until reinstalled.

Action and menu tools return deltas by default: changed units/rules/buttons only. Delta units are [id, name, x, y, direction, properties]; removed lists deleted IDs. Unmentioned state is unchanged. A refresh: "get_board" reply means the agent needs to read the new scene. view="summary", view="board", and view="full" are explicit alternatives. Do not request full state after every move. Replies use one minified JSON text block, without a duplicate structuredContent payload; the example client decodes it.

After upgrading the bridge, restart the game. After changing server code, reconnect/reload the MCP client so it sees the new tools and defaults.

State and action semantics

• Coordinates start at zero. Directions: right = 0, up = 1, left = 2, down = 3.

• get_board().objects[*].at contains [x, y, direction, unit_id] entries. Boards include a format legend; essential instructions are also in tool descriptions, not just server instructions.

• Every reply includes phase, ready, controller, current controlled positions, no_player, can_undo, and a recovery_hint when appropriate. These fields are snapshots, even in delta mode. controller="special" flags parsed LEVEL/EMPTY IS YOU rules; conditions may still prevent control. no_player=1 alone does not imply death on maps or special levels.

• Action replies include requested, executed, stop_reason, and board_changed. executed counts processed commands, including blocked moves, not successful displacement. board_changed compares visible objects, rules, dimensions, and scene, not hidden engine state. A stop reason of control_lost calls for undo/recovery, not further direction commands.

• If the engine reports no player and there is no ordinary controller, act does not send directions/waits: it returns executed=0 and recovery guidance. Map cursors, parsed LEVEL/EMPTY IS YOU rules, and AUTO are exempt. This guard is not a universal death detector. Start a recovery sequence with Z, or call restart_level. A leading undo run is verified before subsequent directions; native recovery-plus-travel may use two engine batches.

• rules contains parsed word triples and their conditions. Do not treat every parsed rule as unconditional. properties reports common checked effects, not every property in the game.

• act uses command(), undo(), code(), and MF_update() inside the real game. It stops early on victory, player death, level transition, pause, or other menu changes. Maps with a SELECT cursor do not stop merely because they have no YOU object.

• undo_depth is the engine's undo buffer depth; turns counts turn-end hooks, including native updates associated with undo. It is not necessarily a puzzle's official move counter.

• won describes the current victory event; wins counts victory hooks since this bridge instance started. Neither is a count of newly unlocked levels or total saved completions.

• During transitions, metadata may already refer to the next level while units still refer to the previous scene. Read fresh state after the transition before planning more moves.

• AUTO/realtime levels require press; they are not deterministic single-step act sessions.

• A request timeout does not prove a move failed. Inspect fresh state before retrying. Mutating requests are never automatically retried, and pending request files are removed.

Architecture

MCP client → stdio server → typed game service → serialized request/ack transport
                                               ↓
                         official Lua mod hooks → native engine functions
                                               ↓
                            native INI telemetry → validated game state

The game's sandbox does not expose Lua io or os. Requests are fixed, allowlisted Lua data tables loaded with the engine's existing dofile. Responses use MF_setfile / MF_store in an isolated INI. The bridge temporarily redirects the native language storage slot and restores lang_<language>.txt immediately afterwards; it does not modify language files. Mods that replace the language storage mechanism may be incompatible.

Request IDs prevent duplicate dispatch within the running bridge. Atomic request replacement, strict response validation, and a bounded cross-process lock protect cooperating clients. Do not run concurrent controllers using different BABA_BRIDGE_DIR values for the same game.

src/baba_is_agent/
  assets/bridge.lua   Game-side adapter (shipped in the wheel)
  config.py          Paths and configuration
  models.py          Validated native state and typed actions
  storage.py         Atomic writes and process locks
  transport.py       Request/acknowledgement protocol
  input.py           Linux uinput and XWayland activation
  installer.py       Reversible installation
  service.py         Gameplay and menu operations
  server.py          MCP tool registration
  cli.py             CLI and diagnostics

Configuration

Requests and responses always live under the game's Data/Languages/baba_mcp/ directory; BABA_BRIDGE_DIR changes only local coordination storage.

Development

uv sync --locked
uv run ruff check .
uv run ruff format --check .
uv run ty check
uv run pytest --cov=baba_is_agent
lua tests/test_bridge.lua src/baba_is_agent/assets/bridge.lua
uv build

Python tests use temporary files, mocked input, and a real MCP stdio subprocess. They do not need Steam, a running game, desktop access, or elevated privileges. The Lua test runs the bridge with io and os removed and checks dispatch deduplication, undo, and error containment. GitHub Actions runs these checks and verifies that the built wheel includes the Lua asset.

An explicitly opt-in live smoke check (read-only):

uv run python scripts/verify_live.py

Ruff uses the repository's Python 3.13 target and 99-character line length. To apply fixes:

uv run ruff check --fix .
uv run ruff format .

Uninstall

uv run baba-mcp uninstall

Restart the game afterwards. This removes only recognized bridge files, not saves or other mods. Remove the MCP client registration separately:

pi mcp remove baba
codex mcp remove baba
claude mcp remove --scope user baba

Run only the command for the client you configured. For project-local Pi registration, add --local; for Claude Code, use the same --scope you used when adding the server. If you configured Codex by editing a TOML file, remove its [mcp_servers.baba] tables from that file.

Security and licensing

This is a local, trusted-client integration, not a remote-control service. There is no HTTP listener, authentication layer, or arbitrary Lua-evaluation MCP tool. A process running as the same desktop user can still tamper with bridge files, and native input is desktop-wide. Only expose these tools to trusted local agents; do not publish the transport directory.

Project code is licensed under MIT. Baba Is You, its executable, scripts, levels, artwork, and trademarks belong to their respective owners and are not included or relicensed by this repository. This is an unofficial project, not affiliated with the game's creators or Steam.