CarlaMCP

Natural language → executable autonomous-vehicle scenarios in the CARLA simulator, exposed to any LLM through the Model Context Protocol.

Built on the tool-based architecture of IROSA, validated against CARLOS for containerized deployment.

A prompt like "an aggressive cut-in from the right at highway speed in heavy rain at night" becomes a validated, reproducible scenario that runs in CARLA — and can be saved as a spec for replay.

How it works

NL prompt ─▶ LLM picks tools ─▶ validated ScenarioManifest ─┬─▶ execute in CARLA ─▶ ScenarioResult
                                (Pydantic primitives)        │
                                                             └─▶ save .json spec ─▶ replay / reuse

Three layers, each independently testable:

Project structure

CarlaMCP/
├── carlamcp/
│   ├── config.py            # connection + simulation/spawn constants
│   ├── enums.py             # CARLAMap, Side, WeatherCondition
│   ├── results.py           # ScenarioResult
│   ├── carla_client.py      # CARLA binding import + get_client()
│   ├── world.py             # map / actor / sync-mode / find_ego helpers
│   ├── storage.py           # save / load scenario specs (export & replay)
│   ├── primitives/          # one file per primitive + the manifest
│   │   ├── cut_in.py  pedestrian.py  weather.py  manifest.py
│   ├── executors/           # one executor per primitive + the orchestrator
│   │   ├── cut_in.py  pedestrian.py  weather.py  manifest.py
│   └── server.py            # FastMCP server + tools + main()
├── tests/                   # pytest suite, runs WITHOUT a simulator
│   ├── fakes.py  conftest.py  test_executor.py  test_primitives.py
├── pyproject.toml  requirements.txt  .gitignore  LICENSE  README.md

Setup

1. Start CARLA (separate terminal)

~/carla/CarlaUE4.sh -RenderOffScreen -quality-level=Low &

2. Install CarlaMCP

cd CarlaMCP
pip install -e ".[dev]"          # package + test deps (no simulator needed)
pip install -e ".[carla,dev]"    # also installs the carla==0.9.15 binding

3. Verify the CARLA connection

python -c "from carlamcp.carla_client import get_client; print(get_client().get_server_version())"

4. Run the MCP server

python -m carlamcp.server          # or just: carlamcp

5. Wire into Claude Desktop

Add to your claude_desktop_config.json (use the absolute path to your clone):

{
  "mcpServers": {
    "carlamcp": {
      "command": "python",
      "args": ["-m", "carlamcp.server"],
      "cwd": "<path-to-your-CarlaMCP-clone>"
    }
  }
}

Restart Claude Desktop; the CarlaMCP tools will appear.

Usage

Once connected, talk to the model:

• "Check if CARLA is running" → get_status

• "Run an aggressive cut-in from the right at 120 km/h with 1.2s TTC" → run_scenario

• "Pedestrian crosses 25 m ahead, occluded by a parked car, in heavy fog"

• "Set the weather to rain_night at intensity 0.8" → set_weather

• "Validate this manifest before running it: {...}" → validate_scenario

Tools

Scenario primitives

Export & replay

Any scenario can be persisted as a validated JSON spec and re-run later — useful for regression sets, sharing, and scenario inpainting / augmentation pipelines:

run_scenario("...", cut_in_speed_kmh=120, cut_in_ttc_s=1.2, cut_in_side="right",
             save_as="rainy_cutin")          # writes scenarios/rainy_cutin.json
replay_scenario("rainy_cutin")               # re-executes the exact same spec

The export layer (carlamcp/storage.py) is deliberately decoupled from execution, so generated specs can be consumed by downstream tooling without a running simulator. See Roadmap → OpenSCENARIO export.

Testing

The suite runs entirely offline — tests/conftest.py injects a fake carla module so the executor logic (spawn geometry, velocities, ordering, error paths, collision metric) is verified without a GPU or a running simulator. This makes it CI-friendly.

pip install -e ".[dev]"
pytest -q

tests/test_executor.py doubles as the canonical example of the project's Google-style (Sphinx Napoleon) docstring convention used across every module.

FAQ — why Python and not C++?

CARLA runs on Unreal Engine (C++), so it's a fair question. For CarlaMCP, C++ buys nothing: the carla PyPI package is a thin binding over the C++ LibCarla client, and everything done here (spawning actors, weather, the traffic manager) is fully exposed in Python with negligible overhead at these tick rates. C++ only becomes worthwhile for work that lives inside the CarlaUE4 server build — custom Unreal sensors/actors/map assets, per-frame high-bandwidth sensor processing where the GIL is the bottleneck, or C++ ScenarioRunner extensions. None of those are on the roadmap, so CarlaMCP stays pure Python.

Roadmap

• Phase 1 — Validated primitives + MCP server + CARLA execution

• Phase 2 — Modular package, offline test suite, scenario export/replay

• Phase 3 — CARLOS Docker integration (docker-compose.yml)

• Phase 4 — CSR/ISR/TCR evaluation harness

• Phase 5 — OpenSCENARIO export for the CARLOS replay pipeline

• Paper — IEEE IV / ITSC submission

Related work

• IROSA — tool-based LLM architecture for robot skill adaptation (DLR/TUM)

• CARLOS — containerized CARLA simulation framework (ika RWTH Aachen)

• SUMO-MCP — MCP for traffic simulation (adjacent domain)

License

MIT.