# webots-mcp

**Give AI agents full control and observability of the [Webots](https://cyberbotics.com) robot simulator.**

An [MCP](https://modelcontextprotocol.io) (Model Context Protocol) server that lets any MCP client — Claude Code, Codex, opencode, or your own agent — launch simulations, train reinforcement-learning models, watch what the robot is doing *while it trains* (without slowing training down), evaluate trained models, and manipulate the scene interactively.

Ask your agent things like:

> *"Start a 200k-step training run and tell me how it's going every few minutes."*
> *"Why did the reward drop? Show me a screenshot of what the robot is doing right now."*
> *"Evaluate the latest model for 10 episodes and compare it with yesterday's run."*
> *"Freeze the sim, put a box 25 cm in front of the robot, step 100 timesteps, and show me the sensor readings."*

## How it works

The Webots simulation state is only reachable from a controller *inside* the simulation loop — and during training, the robot's controller **is** the RL trainer. This server solves that with two independent channels that never touch the training hot path:

```
┌────────────── AI agent (any MCP client) ──────────────┐
│                 webots-mcp  (server.py)                │
└──────┬──────────────────┬──────────────────┬──────────┘
       │ JSON-TCP         │ files            │ subprocess
       ▼                  ▼                  ▼
  OBSERVER robot     runs/*/metrics.jsonl   webots / training
  (webots_bridge.py, (written by the SB3    (start, stop,
   2nd supervisor,    callback in           monitor)
   <extern>)          rl_metrics.py)
       │                  ▲
       ▼                  │
    Webots ◄────── training controller (untouched)
```

1. **Observer robot** — an invisible second supervisor robot (`synchronization FALSE`) runs `webots_bridge.py` as an extern controller. It reads pose, velocity, scene objects and takes screenshots *in parallel with training*, costing ~3% fps.
2. **Metrics files** — a Stable-Baselines3 callback (`rl_metrics.py`) appends one JSON line per episode (with per-component reward breakdown and termination reason) and per policy update (fps, losses). The server only reads files: training never waits for the agent.
3. **Process management** — the server launches and stops Webots instances as subprocesses, each on its own port.

All state lives in files — killing the server loses nothing.

## Quick start

Requirements: Python ≥ 3.10 and [Webots](https://cyberbotics.com) (tested with R2025a on macOS; Linux/Windows install layouts are auto-detected via `WEBOTS_HOME` or `PATH`).

```bash
pip install git+https://github.com/bazucas/Webots_MCP        # installs the `webots-mcp` command
# with the RL extras (stable-baselines3 + gymnasium, for training/eval):
pip install "webots-mcp[rl] @ git+https://github.com/bazucas/Webots_MCP"
```

Or without installing anything, via [uv](https://docs.astral.sh/uv/):

```bash
uvx --from git+https://github.com/bazucas/Webots_MCP webots-mcp
```

Or clone and run from source: `git clone ... && pip install -e ".[dev]"`.

Then register the server in your MCP client — **the working directory must be your Webots project root** (that's how the server finds worlds and `runs/`):

### Claude Code

`.mcp.json` in the project root (cwd is the project automatically):

```json
{
  "mcpServers": {
    "webots": { "command": "webots-mcp" }
  }
}
```

(If you didn't `pip install`, use `"command": "python3", "args": ["/path/to/webots_mcp/webots_mcp/server.py"]` — same for the clients below.)

### opencode

`opencode.json` in the project root:

```json
{
  "mcp": {
    "webots": { "type": "local", "command": ["webots-mcp"] }
  }
}
```

### OpenAI Codex CLI (GPT)

`~/.codex/config.toml` (set `WEBOTS_MCP_PROJECT` since cwd may vary):

```toml
[mcp_servers.webots]
command = "webots-mcp"
env = { WEBOTS_MCP_PROJECT = "/home/you/my-webots-project" }
```

### Cursor

`.cursor/mcp.json` in the project root — same shape as Claude Code's `.mcp.json`.

### Claude Desktop / other MCP clients

Any client that speaks MCP over stdio works: command `webots-mcp` (or `python3 /path/to/webots_mcp/webots_mcp/server.py`). If the client doesn't launch it from your project directory (Claude Desktop doesn't), set `WEBOTS_MCP_PROJECT` in the server's `env`.

Restart/reconnect the client and approve the server — 23 tools appear.

### Configuration

Everything is overridable with environment variables:

| Variable | Meaning | Default |
|---|---|---|
| `WEBOTS_MCP_PROJECT` | project root | current working dir |
| `WEBOTS_HOME` | Webots installation | `/Applications/Webots.app` |
| `WEBOTS_MCP_RL_WORLD` | training world (.wbt) | `<project>/thymio_rl_template/worlds/thymio_rl_env.wbt` |
| `WEBOTS_MCP_EVAL_WORLD` | evaluation world (.wbt) | `.../thymio_eval.wbt` |
| `WEBOTS_MCP_BRIDGE_WORLD` | interactive world (.wbt) | `.../thymio_bridge.wbt` |


## Tools

**Process management**

| Tool | What it does |
|---|---|
| `webots_start(world?, mode?)` | Launch a headless Webots instance; returns port, pid, log path |
| `webots_stop(port)` | Graceful SIGTERM (trainers save models in their `finally`) |
| `webots_list()` | Managed instances and liveness |

**Training**

| Tool | What it does |
|---|---|
| `train_start(total_timesteps?, mode?)` | Start a training run; returns `run_id` |
| `train_status(run_id?)` | timesteps done/target, fps, mean reward, termination breakdown, ETA, alive/finished |
| `metrics_summary(run_id?, last_n?)` | Aggregates of the last N episodes + per-component reward breakdown + first-vs-last trend |
| `metrics_tail(run_id?, n?)` | Raw last lines of `metrics.jsonl` |
| `runs_list()` / `runs_compare(run_ids?)` | All runs; side-by-side comparison |

**Live observation (during training)**

| Tool | What it does |
|---|---|
| `observer_attach(port, robot_name?)` | Connect the bridge — `"observer"` watches a training run; `""` connects to the robot of an interactive world |
| `sim_status()` | Sim time, robot pose and velocity, obstacle inventory (including ones that fell off the arena) |
| `sim_screenshot()` | JPEG of the 3D view, returned inline to the agent |
| `scene_query(def_name)` | Fields of any scene node by DEF |
| `observer_detach()` | Disconnect |

**Evaluation**

| Tool | What it does |
|---|---|
| `eval_run(algo?, episodes?, deterministic?)` | Run a trained model N episodes; returns mean/min/max reward, episode lengths, terminations |

**Interactive control** (world with the robot's controller set to `<extern>`)

| Tool | What it does |
|---|---|
| `sim_get_obs()` / `sim_set_motors(l, r)` | Read sensors / drive motors |
| `sim_pause(bool)` / `sim_step(n)` | Deterministic stepping: freeze → act → step N → read |
| `sim_set_pose()` / `sim_reset_robot()` | Teleport / reset to initial pose |
| `sim_spawn_box(x, y, z?, size?)` / `sim_clear_boxes()` | Ad-hoc obstacles with physics |

Plus the MCP prompt `diagnose_training(run_id?)` — a guided training-health analysis (trend, terminations, reward hacking signals, concrete shaping suggestions) — and resources `webots://runs` and `webots://runs/{run_id}/metrics`.

## Adapting it to your project

The defaults reflect the project this was born in (Thymio II, ISCTE-IUL robotics course), but the pattern is generic. Your project needs:

1. **An observer robot** in the training world (this is the whole trick — copy-paste):

```
Robot {
  name "observer"
  supervisor TRUE
  synchronization FALSE
  controller "<extern>"
}
```

2. **A `DEF ROBOT`** on your main robot node, so the bridge can find it from outside.
3. **The metrics callback** in your SB3 training script:

```python
from rl_metrics import MetricsCallback
model.learn(total_timesteps=n, callback=MetricsCallback(run_name="ppo", runs_root="/path/to/project/runs"))
```

For per-component reward breakdowns, have your env put them in the `info` dict at episode end:

```python
info["episode_metrics"] = {
    "termination_reason": "fall",           # your labels
    "steps": steps, "distance": d, "cells_visited": c,
    "reward_components": {"forward": 12.3, "cliff": -5.0},
}
```

4. For `train_start`, your training controller should read `TRAIN_TIMESTEPS` from the environment; for `eval_run`, your evaluation controller should read `EVAL_ALGO`, `EVAL_EPISODES`, `EVAL_DETERMINISTIC` and write results to the JSON path in `EVAL_OUT`.

`webots_bridge.py` also works standalone without MCP — one JSON command per TCP connection on `127.0.0.1:10101`:

```bash
printf '{"cmd":"get_obs"}\n' | nc -w 5 127.0.0.1 10101
```

## Tests

```bash
pip install -e ".[dev]"
pytest tests -m "not webots"   # unit + MCP stdio protocol (~1 s, no Webots needed)
pytest tests -m webots         # integration against a real Webots (~20 s)
pytest tests                   # everything (37 tests)
```

CI (GitHub Actions) runs the fast suite on Linux, macOS and Windows for every push.

Integration tests need a real Webots project (`WEBOTS_MCP_PROJECT`); they skip cleanly if it's missing. Bridge unit tests inject a fake Webots API via `sys.modules["controller"]`, so most of the suite runs anywhere.

## Limitations & notes

- Webots itself was only exercised on macOS (R2025a). The server code runs on Linux/Windows in CI, and install layouts for all three OSes are covered by unit tests, but end-to-end Webots runs on Linux/Windows are untested — PRs welcome.
- The bridge listens on localhost only, with no authentication — don't expose the port.
- One bridge connection at a time (`BRIDGE_PORT`, default 10101).
- Avoid `simulationReset` in training code (it kills the observer's connection); reset by teleporting the robot instead.

## License

[MIT](LICENSE) — do whatever you want with it.