# mujoco-mcp

MCP server that lets clients load, inspect, and control MuJoCo models, with an optional native interactive window. Multiple independent named sessions, compact JSON responses, deterministic stepping, and real-time viewer playback. Supports stdio and Streamable HTTP. Headless use requires no display or GPU.

## Use from another project (including KaizenHand)

This repository is the **MuJoCo MCP infrastructure**, not the consuming application. It provides physics, contact force/torque, per-step force guarding, seeded conditions/noise, trajectory capture, provenance, and full-state snapshots. The other project owns its MJCF model, control FSM/strategies, retry policy, incident storage, approvals, statistics, dashboard, and physical-rig adapter.

```bash
make install
make server-headless                 # HTTP MCP at http://127.0.0.1:8000/mcp
# If another server is already using 8000:
make server-headless PORT=8001        # http://127.0.0.1:8001/mcp
```

Connect with Streamable HTTP, then load your project's model through `mujoco_load_model(file_path=...)`. No preloaded demo or viewer is required. See the [external-project integration guide](docs/integration.md) for a runnable client, configuration fields, force semantics, trace/snapshot replay, and scope boundaries. Existing servers must be restarted to expose newly added tools; restarting discards their in-memory sessions.

## Connect on this Mac

The project environment is installed in `.venv` with MuJoCo 3.13.0, matching the installed DMG app. The Python package includes its own engine and native viewer. The MCP server opens a window sharing its model and state; it cannot attach to an already-open standalone `MuJoCo.app` window. You can use the same XML/MJB model file in either application.

Start the server in a terminal and leave it running:

```bash
./start.sh --transport streamable-http --port 8000 --viewer --sample pendulum
```

Connect your MCP client with transport **Streamable HTTP** and URL **`http://127.0.0.1:8000/mcp`**. The repository's `.mcp.json` already points to this endpoint for clients that read that format. This is an MCP endpoint, not a browser dashboard. It is reachable from clients on this Mac; a cloud-hosted client cannot reach your Mac's loopback address.

The server starts with session `main`, displaying a paused pendulum. Try these tool calls:

```text
mujoco_list_sessions({})
mujoco_set_running({"session_id":"main","running":true,"ctrl":[0.5]})
mujoco_set_running({"session_id":"main","running":false})
mujoco_step({"session_id":"main","nsteps":100,"ctrl":[0.2]})
mujoco_get_state({"session_id":"main"})
```

Use the MCP play/pause tools for playback. State responses include `viewer_open`, `running`, and `runtime_error`. Closing the window pauses the session but keeps its state available; `mujoco_open_viewer` reopens it. One window per server is supported; close that viewer before opening another session. Reset pauses playback and clears runtime errors. Sessions are in memory and end when the server stops.

Verify the connection without changing the model:

```bash
.venv/bin/python projects/http_client.py
```

To use your own model at startup:

```bash
./start.sh --transport streamable-http --viewer --model /absolute/path/to/robot.xml
```

`Ctrl+C` stops the server and closes its viewer. Restart with the same command when needed; no login/background service is installed.

For clients that only support **stdio**, use `projects/mcp-stdio.json`. That configuration starts its own independent server and viewer when the client connects. Its absolute launcher path is configured for this checkout; update it if you move the project.

The launcher automatically uses `mjpython` on macOS, as required by the [native passive viewer](https://mujoco.readthedocs.io/en/stable/python.html#passive-viewer). The DMG app itself does not need to be running.

## Install

```bash
uv sync --extra dev       # recommended: reproduce the tested environment from uv.lock
# Or install with pip in a Python environment:
pip install .              # runtime
pip install -e ".[dev]"    # development + tests
```

Requires Python >= 3.10; this checkout selects Python 3.12. Dependencies: `mujoco`, `mcp`, `numpy`. The MCP SDK is constrained to v1 because this server uses `FastMCP`; v2 has incompatible APIs.

## Run

```bash
mujoco-mcp                                        # stdio (default)
mujoco-mcp --transport streamable-http --port 8000  # HTTP, binds 127.0.0.1
./start.sh --viewer --sample cartpole               # stdio + native viewer on macOS
./start.sh --transport streamable-http --viewer --play  # HTTP + playing pendulum
```

## Tools

| Tool | Purpose | Key args |
|---|---|---|
| `mujoco_load_model` | Create a session from a model | exactly one of `sample` \| `file_path` \| `xml_string`; optional `session_id` |
| `mujoco_model_info` | Joints, actuators, sensors, geoms, contact pairs, mocap bodies, options | `session_id` |
| `mujoco_reset` | Reset to configured experiment start, model default, or keyframe; clear stop/error | `session_id`, `keyframe` (name or index) |
| `mujoco_step` | Advance physics, optionally setting ctrl first | `session_id`, `nsteps` (1–100000), `ctrl`, `include`, `precision` |
| `mujoco_get_state` | Read state without stepping | `session_id`, `include`, `precision` |
| `mujoco_set_state` | Overwrite qpos / qvel / ctrl / time, then `mj_forward` | `session_id`, `qpos`, `qvel`, `ctrl`, `time` |
| `mujoco_list_sessions` | Enumerate active sessions | — |
| `mujoco_close_session` | Free a session | `session_id` |
| `mujoco_open_viewer` | Display a session in a native MuJoCo window | `session_id` |
| `mujoco_close_viewer` | Close the window and pause, retaining the model/state | `session_id` |
| `mujoco_set_running` | Play/pause the visible session, optionally update controls | `session_id`, `running`, `ctrl` |
| `mujoco_get_wrench` | All external rigid contacts on a body subtree; world force and torque | `session_id`, `body_name` |
| `mujoco_step_guarded` | Step with force-limit stop and optional trajectory recording | `session_id`, `body_name`, `force_limit` (N), `nsteps` (1–10000), `ctrl`, `record_every` |
| `mujoco_configure_experiment` | Set seeded conditions, friction, bias and observation noise; reset | `session_id`, `config` |
| `mujoco_get_provenance` | Compiled-model/config/code hashes, versions, source label | `session_id` |
| `mujoco_get_snapshot` | Capture unrounded full integration state for continuation | `session_id` |
| `mujoco_restore_snapshot` | Restore matching model/config/engine state and guard status | `session_id`, `snapshot` |

Conventions:

- Responses are compact JSON strings. `ctrl` follows actuator order. For `qpos`/`qvel`, use joint types and `qpos_adr`/`dof_adr` from `mujoco_model_info`: free and ball joints occupy multiple entries.
- `ctrl` is either a full vector in actuator order or `{actuator_name: value}` for a subset; it persists on the session until changed.
- `include` selects state blocks: `qpos`, `qvel`, `ctrl`, `sensors` (the default four), `sensors_raw` (without configured observation noise), `bodies` (world-frame pos/quat), `contacts` (geom pairs, penetration depth, normal force; capped at 20). Wrench/guard calculations use all contacts, not this display cap.
- `precision` (default 5) rounds state views. Guarded traces use 9 decimals for state; wrench values and snapshots are unrounded. Use snapshots for exact continuation, not rounded `get_state` results.
- Batch physics into one call: `nsteps=500` is 1 s at the default 0.002 timestep. Avoid step-per-call loops.
- Pause real-time playback before calling `mujoco_step`. Its returned sensors/body poses are recomputed at the final state. Use `mujoco_set_running(running=true, ctrl=...)` to update controls during playback.
- Force limits apply only during `mujoco_step_guarded`, not ordinary stepping/playback. Once tripped, the stop blocks every stepping path until reset (including experiment reconfiguration) or restoration of a pre-stop snapshot. Detection is at physics-step boundaries and does not guarantee zero overshoot or physical-robot safety.

## Bundled samples

| Sample | System | Actuators | Keyframes |
|---|---|---|---|
| `pendulum` | torque-limited pendulum | `hinge_motor` | `down`, `up` |
| `cartpole` | cart with unactuated pole | `slide_motor` | `balanced`, `tilted` |
| `arm2` | planar 2-link arm (gravity-free) with tip site sensor | `shoulder_motor`, `elbow_motor` | — |
| `box_drop` | free box falling onto a plane (contact demo) | — | — |

## Embedding in your orchestrator

`projects/agent_client.py` is a complete working loop: it spawns the server over stdio, loads the cartpole, and balances the pole from sensor feedback through nothing but MCP tool calls. Core pattern:

```python
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client

params = StdioServerParameters(command="mujoco-mcp")
async with stdio_client(params) as (read, write):
    async with ClientSession(read, write) as session:
        await session.initialize()
        tools = (await session.list_tools()).tools
        result = await session.call_tool(
            "mujoco_step", {"session_id": "sim1", "nsteps": 500, "ctrl": [0.3]}
        )
```

To surface the tools to a Claude (or any tool-calling) model, map them 1:1 and route `tool_use` back through the session:

```python
anthropic_tools = [
    {"name": t.name, "description": t.description, "input_schema": t.inputSchema}
    for t in tools
]

result = await session.call_tool(block.name, block.input)
tool_result_content = result.content[0].text
```

Errors come back with `isError=true` and an actionable message (valid session ids, actuator order, allowed values), so they are safe to feed straight back to the model as `tool_result` with `is_error=true`.

## Loading your own models

- `file_path`: any MJCF `.xml` (relative includes/assets resolve normally) or compiled `.mjb`, resolved on the machine running the server.
- MuJoCo Menagerie: `git clone https://github.com/google-deepmind/mujoco_menagerie`, then `file_path=".../mujoco_menagerie/franka_emika_panda/scene.xml"`.
- `xml_string`: for models generated on the fly by the agent.

## Configuration

| Env var | Default | Effect |
|---|---|---|
| `MUJOCO_MCP_MAX_SESSIONS` | 16 | Max concurrent sessions |
| `MUJOCO_MCP_MODEL_ROOTS` | unset (any path) | Colon-separated allowlist of directories `file_path` may load from |

## Claude Code / Claude Desktop

For Claude Code, the project's `.mcp.json` connects to the running HTTP server:

```json
{"mcpServers":{"mujoco":{"type":"http","url":"http://127.0.0.1:8000/mcp"}}}
```

For stdio clients such as Claude Desktop, copy the `mujoco` entry from `projects/mcp-stdio.json` into the client's MCP configuration. The client then launches its own server; the manually started HTTP server is not used. The launcher uses absolute paths, so it works even if the client starts in a different directory.

## Notes

- Deterministic manual stepping: same model + same reset + same ctrl/step sequence reproduces the trajectory. Real-time playback uses wall-clock scheduling and is not a deterministic control interface.
- Divergence (non-finite state) is detected after stepping and returned as an error with recovery guidance instead of NaN payloads.
- State lives in server memory; run one server process per trusted consumer. stdio has no auth; streamable HTTP binds `127.0.0.1` — put a reverse proxy with auth in front before exposing it.

## Tests

```bash
.venv/bin/python -m pytest
# Opens actual native windows; requires a logged-in desktop:
MUJOCO_MCP_TEST_VIEWER=1 .venv/bin/python -m pytest tests/test_viewer_integration.py
```

Covers every tool, error paths (unknown session/actuator/keyframe, wrong vector lengths, divergence, path allowlist), determinism, contact-force sanity, and an end-to-end stdio client session. Viewer checks cover shared state, real-time play/pause, window closure, session cleanup, and automatic pause on instability. The opt-in GUI integration test exercises the actual native window over MCP, including closing and reopening it.

Experiment tests cover seeded noise, atomic configuration, geom/pair friction, actuator bias, mocap poses, full-state snapshot continuation, non-cancelling contact loads, force-stop latching, and bounded trace sampling. Both stdio and an isolated HTTP server are exercised. A 100-trial headless box-drop batch checks transport and physics plumbing; it is **not** evidence that an external connector model or recovery strategy has been validated.