Xbox Agent Bridge
Uses Tailscale networking, often via a Raspberry Pi subnet router, to make an Xbox on another network reachable from the Windows PC running Greenlight for remote play.
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
Xbox Agent Bridge
Give AI agents eyes and a controller for your Xbox. Research the game, plan a mission, play through Greenlight, and carry useful experience into the next session.
Xbox Agent Bridge connects a vision-capable agent to a Windows PC running Greenlight. The agent sees the screen and uses sticks, triggers and buttons. A Raspberry Pi running Tailscale can make an Xbox on another network reachable from that PC.
Status: experimental alpha. The simulation, API and MCP loop are testable without an Xbox. Reliable autonomous gameplay and mission completion have not been demonstrated by this project. A model or an MCP-capable agent supplies the reasoning; this repository supplies observation, control, memory and evaluation tools.
What this enables
Play through the normal interface: observe the streamed screen and send complete controller reports, including simultaneous movement, looking and triggers.
Research before playing: use your agent's web tools, then save game-specific research with source URLs.
Work toward missions: lease the controller for a goal, record progress against an observation, and report an outcome.
Remember what works: retain strategies, controller calibration notes and reusable skill recipes across sessions.
Measure the control loop: record host action timing, interruptions and agent-reported mission outcomes. Simulation data is separated from live data.
Keep human control: interrupt with Stop, release all inputs, and disconnect Greenlight to return to a physical controller.
Tailscale supplies network reachability, Greenlight supplies Remote Play, and this bridge supplies the agent interface. This does not install an agent on the Xbox, expose game memory, or require Xbox Developer Mode.
Related MCP server: local-game-mcp
Architecture
flowchart LR
A[Vision-capable agent] -->|MCP or HTTP over SSH| B[Windows: Xbox Agent Bridge]
B -->|XInput controller| G[Greenlight]
G -->|client-area screen capture| B
G <-->|Remote Play| P[Pi 5: Tailscale subnet router]
P <-->|local network| X[Xbox]
A <-->|research, skills, mission evaluations| K[(Local game memory)]
B --> KOn a local network, the PC can reach the Xbox directly and the Pi is optional. The recommended remote arrangement keeps the controller executor beside Greenlight and tunnels the small agent API over SSH. For fast feedback, place the vision policy there too.
Try it without an Xbox
Python 3.11+ is required. No API key, account or controller driver is needed for the demo.
git clone https://github.com/kay2stack/xbox-agent-bridge.git
cd xbox-agent-bridge
python -m venv .venv
.venv\Scripts\python -m pip install -e ".[mcp]"
.venv\Scripts\xbox-agent serve --demoOpen http://127.0.0.1:39420. In a second terminal:
.venv\Scripts\xbox-agent tokenPaste that token into the dashboard. It is kept in the tab's memory. Lease the controller and capture a synthetic frame, or run the demonstration loop:
.venv\Scripts\python examples\demo_agent.pyThe demo creates no input device and refuses to run against a live bridge. On macOS/Linux use .venv/bin/python and .venv/bin/xbox-agent.
Connect your real Xbox
Follow Windows and Greenlight setup, and Pi/Tailscale networking if the console is remote. The bridge starts disarmed, with a neutral virtual controller. Open Greenlight after starting the bridge so it discovers that controller, authenticate personally, then connect the console.
Live control uses vgamepad and an existing ViGEmBus driver. ViGEmBus is archived. Driver installation is a separate, deliberate setup step; the bridge does not install it. Disable vgamepad's bundled driver installer when adding the controller extra:
$env:VGAMEPAD_SKIP_VIGEMBUS_INSTALL = 'true'
.venv\Scripts\python -m pip install -e ".[controller,mcp]"
.venv\Scripts\xbox-agent doctor
.venv\Scripts\xbox-agent serveOne Greenlight window must be visible and in the foreground. The bridge captures its client area, checks the executable name, and refuses input when focus changes. See the setup guide for controller detection and the required in-game acceptance checks.
Give an agent the tools
Add this to your MCP client's server configuration, replacing the executable with the absolute path to your virtual environment:
{
"mcpServers": {
"xbox": {
"command": "C:\\path\\to\\xbox-agent-bridge\\.venv\\Scripts\\xbox-agent.exe",
"args": ["mcp"]
}
}
}The bridge must already be running under the same Windows user. The MCP adapter reads the local token file. For an agent on another host, see the SSH tunnel instructions and set XBOX_AGENT_TOKEN there.
Tool | Purpose |
| Check live/demo mode, controller lease and faults |
| Read research, skills, outcomes and timing metrics |
| Save sourced research, strategies, skills or calibration |
| Lease the controller for a game and mission |
| Get a screen image and observation ID |
| Apply one short, complete controller report, then neutral |
| Record an observation-backed evaluation; terminal outcomes stop control |
| Release inputs and revoke the session |
Use the agent playbook as the agent's task instructions. The HTTP API supports other frameworks without MCP.
How improvement works
The intended loop is research → recall → plan → observe → act → evaluate → remember. The agent can compare outcomes, keep successful strategies, and tune its next decisions using saved evidence. Skill notes are decision recipes; they are never blindly replayed as long macros.
This is persistent experience and measurement, not automatic model training. Recorded success is an agent's assessment, not authoritative game telemetry. Timing metrics measure the host's controller execution, not network latency or frame-to-input response. Fast aiming, racing and combat will need a low-latency policy and game-specific evaluation; see the roadmap.
Development
.venv\Scripts\python -m pip install -e ".[mcp,dev]"
.venv\Scripts\python -m unittest discover -s tests -v
.venv\Scripts\python -m ruff check .
.venv\Scripts\python -m buildTests use fake controllers and synthetic images. CI runs on Windows and Linux. See CONTRIBUTING.md for useful first contributions and SECURITY.md for the trust boundary.
MIT licensed. Independent community project; not affiliated with Microsoft, Xbox, Tailscale or Greenlight. Third-party software remains under its own licenses. See THIRD_PARTY.md.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
This server cannot be deployed
Maintenance
Related MCP Connectors
Human-in-the-loop control panel for AI agents: start runs, review them, answer their requests.
Run your Krtuworks apps from an AI agent: setup, remote config, stats, players, bug reports.
Configure AI agents, give them knowledge and tools, and read back every generation they run.
Let AI agents use your real, signed-in Chrome: tabs, clicks, forms, screenshots and video.
Related MCP Servers
- AlicenseNot gradedqualityBmaintenanceEnables AI agents to remotely control gaming PCs for automated gameplay with tools for screen capture, mouse/keyboard control, workflows, and system operations.2MIT
- FlicenseNot gradedqualityDmaintenanceEnables AI agents to play local Windows games through low-level keyboard/mouse input, screen capture, OCR, and per-game profiles for semantic actions.-
- AlicenseNot gradedqualityAmaintenanceRealtime PC eyes and hands for any AI agent — provides desktop, browser, file, registry, and OS control via batched execution and lease-based safety gates.MIT
- AlicenseNot gradedqualityBmaintenanceEnables any local MCP agent host to observe and operate a real Stardew Valley game through 28 tools covering world snapshots, screenshots, movement, inventory, menus, buying/shipping, containers, crafting, placing, fishing, combat and save loading. Writes run only through normal in-game flows under a single-writer control lease with ledger-based idempotency, epoch invalidation, fail-safe release on disconnect, and human takeover via web console or hotkey.1MIT