# CarlaMCP

Natural language → executable autonomous-vehicle scenarios in the
[CARLA](https://carla.org/) simulator, exposed to any LLM through the
[Model Context Protocol](https://modelcontextprotocol.io/).

Built on the tool-based architecture of [IROSA](https://arxiv.org/abs/2603.03897),
validated against [CARLOS](https://github.com/ika-rwth-aachen/carlos) for
containerized deployment.

A prompt like *"an aggressive cut-in from the right at highway speed in heavy
rain at night"* becomes a validated, reproducible scenario that runs in CARLA —
and can be saved as a spec for replay.

---

## How it works

```
NL prompt ─▶ LLM picks tools ─▶ validated ScenarioManifest ─┬─▶ execute in CARLA ─▶ ScenarioResult
                                (Pydantic primitives)        │
                                                             └─▶ save .json spec ─▶ replay / reuse
```

Three layers, each independently testable:

| Layer | Package | Responsibility |
|---|---|---|
| **Primitives** | `carlamcp/primitives/` | Pydantic models with LLM-facing descriptions, range constraints, and cross-field validators. The LLM never writes raw simulator code. |
| **Executors** | `carlamcp/executors/` | Translate validated primitives into `carla` API calls. The only layer that touches the live simulator. |
| **Server** | `carlamcp/server.py` | The MCP server exposing tools to the model. |

## Project structure

```
CarlaMCP/
├── carlamcp/
│   ├── config.py            # connection + simulation/spawn constants
│   ├── enums.py             # CARLAMap, Side, WeatherCondition
│   ├── results.py           # ScenarioResult
│   ├── carla_client.py      # CARLA binding import + get_client()
│   ├── world.py             # map / actor / sync-mode / find_ego helpers
│   ├── storage.py           # save / load scenario specs (export & replay)
│   ├── primitives/          # one file per primitive + the manifest
│   │   ├── cut_in.py  pedestrian.py  weather.py  manifest.py
│   ├── executors/           # one executor per primitive + the orchestrator
│   │   ├── cut_in.py  pedestrian.py  weather.py  manifest.py
│   └── server.py            # FastMCP server + tools + main()
├── tests/                   # pytest suite, runs WITHOUT a simulator
│   ├── fakes.py  conftest.py  test_executor.py  test_primitives.py
├── pyproject.toml  requirements.txt  .gitignore  LICENSE  README.md
```

---

## Setup

### 1. Start CARLA (separate terminal)

```bash
~/carla/CarlaUE4.sh -RenderOffScreen -quality-level=Low &
```

### 2. Install CarlaMCP

```bash
cd CarlaMCP
pip install -e ".[dev]"          # package + test deps (no simulator needed)
pip install -e ".[carla,dev]"    # also installs the carla==0.9.15 binding
```

### 3. Verify the CARLA connection

```bash
python -c "from carlamcp.carla_client import get_client; print(get_client().get_server_version())"
```

### 4. Run the MCP server

```bash
python -m carlamcp.server          # or just: carlamcp
```

### 5. Wire into Claude Desktop

Add to your `claude_desktop_config.json` (use the absolute path to *your* clone):

```json
{
  "mcpServers": {
    "carlamcp": {
      "command": "python",
      "args": ["-m", "carlamcp.server"],
      "cwd": "<path-to-your-CarlaMCP-clone>"
    }
  }
}
```

Restart Claude Desktop; the CarlaMCP tools will appear.

---

## Usage

Once connected, talk to the model:

- *"Check if CARLA is running"* → `get_status`
- *"Run an aggressive cut-in from the right at 120 km/h with 1.2s TTC"* → `run_scenario`
- *"Pedestrian crosses 25 m ahead, occluded by a parked car, in heavy fog"*
- *"Set the weather to rain_night at intensity 0.8"* → `set_weather`
- *"Validate this manifest before running it: {...}"* → `validate_scenario`

### Tools

| Tool | Purpose |
|---|---|
| `get_status` | Check the simulator is reachable. |
| `list_maps` | List supported towns and their characteristics. |
| `run_scenario` | Build + validate + execute a scenario (optional `save_as`). |
| `set_weather` | Apply a weather preset only. |
| `validate_scenario` | Validate a manifest JSON without running it. |
| `save_scenario` | Validate and persist a manifest for later. |
| `replay_scenario` | Load a saved spec and execute it. |

### Scenario primitives

| Primitive | Key parameters | Default map |
|---|---|---|
| `CutInManeuver` | `speed_kmh`, `ttc_s`, `side`, `ego_speed_kmh` | Town04 (motorway) |
| `PedestrianCross` | `gap_acceptance_s`, `crossing_distance_m`, `occluded` | Town03 (urban) |
| `WeatherOverlay` | `condition`, `intensity` | Any |

### Export & replay

Any scenario can be persisted as a validated JSON spec and re-run later — useful
for regression sets, sharing, and scenario inpainting / augmentation pipelines:

```text
run_scenario("...", cut_in_speed_kmh=120, cut_in_ttc_s=1.2, cut_in_side="right",
             save_as="rainy_cutin")          # writes scenarios/rainy_cutin.json
replay_scenario("rainy_cutin")               # re-executes the exact same spec
```

The export layer (`carlamcp/storage.py`) is deliberately decoupled from
execution, so generated specs can be consumed by downstream tooling without a
running simulator. See **Roadmap → OpenSCENARIO export**.

---

## Testing

The suite runs entirely offline — `tests/conftest.py` injects a fake `carla`
module so the executor logic (spawn geometry, velocities, ordering, error paths,
collision metric) is verified without a GPU or a running simulator. This makes
it CI-friendly.

```bash
pip install -e ".[dev]"
pytest -q
```

`tests/test_executor.py` doubles as the canonical example of the project's
Google-style (Sphinx [Napoleon](https://sphinxcontrib-napoleon.readthedocs.io/))
docstring convention used across every module.

---

## FAQ — why Python and not C++?

CARLA runs on Unreal Engine (C++), so it's a fair question. For CarlaMCP, C++
buys nothing: the `carla` PyPI package is a thin binding over the C++
`LibCarla` client, and everything done here (spawning actors, weather, the
traffic manager) is fully exposed in Python with negligible overhead at these
tick rates. C++ only becomes worthwhile for work that lives *inside the CarlaUE4
server build* — custom Unreal sensors/actors/map assets, per-frame
high-bandwidth sensor processing where the GIL is the bottleneck, or C++
ScenarioRunner extensions. None of those are on the roadmap, so CarlaMCP stays
pure Python.

---

## Roadmap

- [x] Phase 1 — Validated primitives + MCP server + CARLA execution
- [x] Phase 2 — Modular package, offline test suite, scenario export/replay
- [ ] Phase 3 — CARLOS Docker integration (`docker-compose.yml`)
- [ ] Phase 4 — CSR/ISR/TCR evaluation harness
- [ ] Phase 5 — OpenSCENARIO export for the CARLOS replay pipeline
- [ ] Paper — IEEE IV / ITSC submission

## Related work

- [IROSA](https://arxiv.org/abs/2603.03897) — tool-based LLM architecture for robot skill adaptation (DLR/TUM)
- [CARLOS](https://github.com/ika-rwth-aachen/carlos) — containerized CARLA simulation framework (ika RWTH Aachen)
- [SUMO-MCP](https://arxiv.org/abs/2506.03548) — MCP for traffic simulation (adjacent domain)

## License

[MIT](LICENSE).