Xbox Agent Bridge
by kay2stack
README.md
# Xbox Agent Bridge
**Give AI agents eyes and a controller for your Xbox.** Research the game, plan a mission, play through Greenlight, and carry useful experience into the next session.
Xbox Agent Bridge connects a vision-capable agent to a Windows PC running [Greenlight](https://github.com/unknownskl/greenlight). The agent sees the screen and uses sticks, triggers and buttons. A Raspberry Pi running Tailscale can make an Xbox on another network reachable from that PC.
**Status: experimental alpha.** The simulation, API and MCP loop are testable without an Xbox. Reliable autonomous gameplay and mission completion have **not** been demonstrated by this project. A model or an MCP-capable agent supplies the reasoning; this repository supplies observation, control, memory and evaluation tools.
## What this enables
- **Play through the normal interface:** observe the streamed screen and send complete controller reports, including simultaneous movement, looking and triggers.
- **Research before playing:** use your agent's web tools, then save game-specific research with source URLs.
- **Work toward missions:** lease the controller for a goal, record progress against an observation, and report an outcome.
- **Remember what works:** retain strategies, controller calibration notes and reusable skill recipes across sessions.
- **Measure the control loop:** record host action timing, interruptions and agent-reported mission outcomes. Simulation data is separated from live data.
- **Keep human control:** interrupt with Stop, release all inputs, and disconnect Greenlight to return to a physical controller.
Tailscale supplies network reachability, Greenlight supplies Remote Play, and this bridge supplies the agent interface. This does not install an agent on the Xbox, expose game memory, or require Xbox Developer Mode.
## Architecture
```mermaid
flowchart LR
A[Vision-capable agent] -->|MCP or HTTP over SSH| B[Windows: Xbox Agent Bridge]
B -->|XInput controller| G[Greenlight]
G -->|client-area screen capture| B
G <-->|Remote Play| P[Pi 5: Tailscale subnet router]
P <-->|local network| X[Xbox]
A <-->|research, skills, mission evaluations| K[(Local game memory)]
B --> K
```
On a local network, the PC can reach the Xbox directly and the Pi is optional. The recommended remote arrangement keeps the controller executor beside Greenlight and tunnels the small agent API over SSH. For fast feedback, place the vision policy there too.
## Try it without an Xbox
Python 3.11+ is required. No API key, account or controller driver is needed for the demo.
```powershell
git clone https://github.com/kay2stack/xbox-agent-bridge.git
cd xbox-agent-bridge
python -m venv .venv
.venv\Scripts\python -m pip install -e ".[mcp]"
.venv\Scripts\xbox-agent serve --demo
```
Open **http://127.0.0.1:39420**. In a second terminal:
```powershell
.venv\Scripts\xbox-agent token
```
Paste that token into the dashboard. It is kept in the tab's memory. Lease the controller and capture a synthetic frame, or run the demonstration loop:
```powershell
.venv\Scripts\python examples\demo_agent.py
```
The demo creates no input device and refuses to run against a live bridge. On macOS/Linux use `.venv/bin/python` and `.venv/bin/xbox-agent`.
## Connect your real Xbox
Follow [Windows and Greenlight setup](docs/windows-setup.md), and [Pi/Tailscale networking](docs/networking.md) if the console is remote. The bridge starts **disarmed**, with a neutral virtual controller. Open Greenlight after starting the bridge so it discovers that controller, authenticate personally, then connect the console.
Live control uses vgamepad and an existing ViGEmBus driver. **ViGEmBus is archived.** Driver installation is a separate, deliberate setup step; the bridge does not install it. Disable vgamepad's bundled driver installer when adding the controller extra:
```powershell
$env:VGAMEPAD_SKIP_VIGEMBUS_INSTALL = 'true'
.venv\Scripts\python -m pip install -e ".[controller,mcp]"
.venv\Scripts\xbox-agent doctor
.venv\Scripts\xbox-agent serve
```
One Greenlight window must be visible and in the foreground. The bridge captures its client area, checks the executable name, and refuses input when focus changes. See the setup guide for controller detection and the required in-game acceptance checks.
## Give an agent the tools
Add this to your MCP client's server configuration, replacing the executable with the absolute path to your virtual environment:
```json
{
"mcpServers": {
"xbox": {
"command": "C:\\path\\to\\xbox-agent-bridge\\.venv\\Scripts\\xbox-agent.exe",
"args": ["mcp"]
}
}
}
```
The bridge must already be running under the same Windows user. The MCP adapter reads the local token file. For an agent on another host, see the SSH tunnel instructions and set `XBOX_AGENT_TOKEN` there.
| Tool | Purpose |
| --- | --- |
| `xbox_status` | Check live/demo mode, controller lease and faults |
| `xbox_recall_game` | Read research, skills, outcomes and timing metrics |
| `xbox_remember` | Save sourced research, strategies, skills or calibration |
| `xbox_acquire` | Lease the controller for a game and mission |
| `xbox_observe` | Get a screen image and observation ID |
| `xbox_act` | Apply one short, complete controller report, then neutral |
| `xbox_report_progress` | Record an observation-backed evaluation; terminal outcomes stop control |
| `xbox_stop` | Release inputs and revoke the session |
Use the [agent playbook](docs/agent-playbook.md) as the agent's task instructions. The [HTTP API](docs/api.md) supports other frameworks without MCP.
## How improvement works
The intended loop is **research → recall → plan → observe → act → evaluate → remember**. The agent can compare outcomes, keep successful strategies, and tune its next decisions using saved evidence. Skill notes are decision recipes; they are never blindly replayed as long macros.
This is persistent experience and measurement, not automatic model training. Recorded success is an agent's assessment, not authoritative game telemetry. Timing metrics measure the host's controller execution, not network latency or frame-to-input response. Fast aiming, racing and combat will need a low-latency policy and game-specific evaluation; see [the roadmap](docs/roadmap.md).
## Development
```powershell
.venv\Scripts\python -m pip install -e ".[mcp,dev]"
.venv\Scripts\python -m unittest discover -s tests -v
.venv\Scripts\python -m ruff check .
.venv\Scripts\python -m build
```
Tests use fake controllers and synthetic images. CI runs on Windows and Linux. See [CONTRIBUTING.md](CONTRIBUTING.md) for useful first contributions and [SECURITY.md](SECURITY.md) for the trust boundary.
MIT licensed. Independent community project; not affiliated with Microsoft, Xbox, Tailscale or Greenlight. Third-party software remains under its own licenses. See [THIRD_PARTY.md](THIRD_PARTY.md).
This server cannot be deployed
Maintenance
ActivityMaintained
ResponsivenessNo recent community issues