dex-isaac-mcp
Summary: An MCP server that lets an AI agent drive a live, persistent Isaac Sim session (Kit stays up in a Docker daemon) and launch Isaac Lab training runs.
Session:
sim_status,sim_up,sim_down,sim_reload— start/stop/restart the daemon (headless or GUI, robot config or raw USD, optional ground).Inspect:
sim_inspect_joints(USD articulation DOFs, loop closures, roots),sim_get_joint_state(positions/velocities, normalized targets, limits),sim_get_stats(travel and tracking error).Drive:
sim_set_targets(normalized 0..1 vectors or per-joint),sim_list_poses/sim_set_pose(named poses with blending),sim_step,sim_play,sim_wave.Diagnose:
sim_range_test(find blocked joints below 0.9 travel),sim_sweep(try several values of one live parameter and restore the original).Tune live:
sim_set_params(stiffness, damping, effort) andsim_set_coupling(software mimic ratios).Scene:
sim_spawn_object(primitives or USD props, static/kinematic),sim_list_objects,sim_remove_object.View and capture:
sim_set_camera,sim_screenshot(GUI),sim_frame_robot,sim_set_capture_camera,sim_set_backdrop,sim_capture, and captioned GIF recording viasim_record_start/sim_record_caption/sim_record_stop— all headless.Train:
train_start(headless Isaac Lab run in its own container),train_list,train_status,train_logs,train_metrics(TensorBoard scalars),train_checkpoints,train_stop; runs survive client disconnect.Any robot: point it at a USD plus a small JSON config (joints, poses, couplings, gains, solver, camera); own assets mountable via a compose override.
Drives a live, persistent NVIDIA Isaac Sim (Kit) session — starting/stopping the daemon, inspecting articulation joints and state, setting normalized joint targets and named poses, stepping/playing physics, capturing screenshots and tuning gains and couplings — and launches and monitors NVIDIA Isaac Lab reinforcement-learning training runs, including their status, logs, TensorBoard scalars and checkpoints.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@dex-isaac-mcp start the sim with the Franka, move it to ready, and take a screenshot"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
dex-isaac-mcp

Recorded headless with the server's own tools (sim_frame_robot, sim_set_backdrop, sim_record_start), driving the Allegro hand example. Each caption is the request and the tool call it became.
An MCP server that lets an AI agent (Claude Code, or any MCP client) drive a live, persistent Isaac Sim session and launch Isaac Lab training runs.
Kit takes tens of seconds to boot. If every experiment is a fresh launch, most of your time goes to waiting. Here Kit starts once inside a daemon and stays up. Each tool call lands in that running session, so changing a gain, stepping physics or taking a screenshot costs a frame, not a relaunch.
MCP client ──stdio──▶ dex_isaac_mcp (host, plain python)
│ newline-delimited JSON over a Unix socket
▼
scripts/simd.py (Isaac Lab container, Kit stays up)
└─ one articulation, described by a robot JSONAny articulated robot. Point it at a USD and a small JSON config. Franka and Allegro examples are included.
Normalized joint control. Targets are
0..1, where 0 is a joint's lower limit and 1 its upper limit, so agents don't need to know radians or meters.Named poses per robot (
home,fist, …), which can be blended part-way.Headless capture and GIF recording from an auto-framed camera, with captions. The clip above was made this way.
Props: spawn a table, a ball or a USD into the running scene and read back where they settle.
Measurements: joint state, per-joint travel and tracking error, a range test that finds blocked joints, and parameter sweeps inside one session.
Training control: each run is a detached
docker compose run. You can poll its status, TensorBoard scalars, checkpoints and logs.
How it differs from other Isaac Sim MCP servers
NVIDIA's official Isaac Sim MCP is a documentation search for coding assistants, and works well alongside this one. Servers like isaacsim-mcp-server and omni-mcp/isaac-sim-mcp run inside the Isaac Sim GUI and cover scene building broadly. This one runs headless, as a daemon in Docker, and focuses on tuning, measuring and training one robot.
NVIDIA Isaac Sim MCP | In-GUI servers ( | dex-isaac-mcp | |
What it is | Docs and code search | Kit extension inside a running Isaac Sim | External daemon; Kit stays up in a container |
Controls the simulation | No | Yes | Yes |
Headless / remote GPU box | n/a | Needs the GUI | Yes: headless capture, Docker, Unix socket |
Scene building (lights, materials, sensors, asset library) | n/a | Broad | Minimal: primitive and USD props |
Robot setup | n/a | Built-in robot library | Any USD plus a small JSON config (joints, poses, couplings, gains) |
Measurement | n/a | Joint and prim state | Range tests that find blocked joints, tracking error, sweeps that restore the original value |
Live tuning | n/a | Not documented | PD gains, effort, software mimic couplings |
Isaac Lab training | n/a | Not documented | Launch, poll, TensorBoard metrics, checkpoints, stop |
Recording | n/a | Camera captures | Captioned GIFs from an auto-framed camera, headless |
Platform | Any | Wherever Isaac Sim runs | Linux, NVIDIA GPU, Docker, NGC |
Pick an in-GUI server to build a scene by talking to it, interactively, with a wide robot and asset library.
Pick this one to have an agent tune, debug and train a specific robot (your own, from a USD), unattended or on a remote box, with numbers it can act on instead of only screenshots.
Related MCP server: IsaacSim-MCP
Requirements
Linux with an NVIDIA GPU that Isaac Sim supports, plus the NVIDIA Container Toolkit
Docker with the Compose plugin
An NGC login to pull the Isaac Lab image:
docker login nvcr.io(user$oauthtoken, password: your NGC API key)Python ≥ 3.10 on the host, for the MCP server only
Quick start
git clone <this repo> dex-isaac-mcp && cd dex-isaac-mcp
# 1. Build the image (Isaac Lab 2.3.2 base, pinned by digest)
cd docker && docker compose build && cd ..
# 2. Install the host-side server, editable so it finds this clone
pip install -e . # add [metrics] for train_metrics: pip install -e '.[metrics]'
# 3. Register it with Claude Code
claude mcp add isaac -- python -m dex_isaac_mcp
# 4. Allow the container to open windows (once per login; needed for screenshots)
xhost +local:dockerThen ask the agent something like "start the sim with the Franka, move it to ready and show me a screenshot." It will call sim_up, sim_set_pose, sim_step and sim_screenshot.
From PyPI instead: pip install dex-isaac-mcp. The daemon still runs from a clone (it needs docker/ and scripts/simd.py), so point the server at it:
claude mcp add isaac -e ISAAC_MCP_HOME=/abs/path/dex-isaac-mcp -- dex-isaac-mcpOther MCP clients can launch python -m dex_isaac_mcp (or the dex-isaac-mcp script) over stdio.
The first sim_up takes several minutes: Kit builds its shader cache and downloads Nucleus assets. Later starts are much faster.
Running the daemon by hand
sim_up deletes its container (--rm) when the daemon exits, so a crash on startup takes its traceback with it. To see the error, run the daemon in the foreground:
cd docker
docker compose run --rm simd scripts/simd.py --gui --robot examples/robots/franka.json
docker compose run --rm simd scripts/simd.py --headless --usd /path/in/container/robot.usd
docker compose run --rm simd scripts/simd.py --sliders --robot examples/robots/allegro_hand.json--sliders opens an omni.ui panel with one slider per driven joint. The socket stays live alongside it.
Tools
Session
Tool | What it does |
| Whether the daemon is up, plus its robot, driven joints, poses, couplings and gains |
| Start the daemon container ( |
| Stop the daemon |
| Restart with a different spawn property: robot, USD, |
| Undo an experiment in a few frames: remove props, restore spawn gains and couplings, write joints and root back to their spawn state. |
Inspect
Tool | What it does |
| A USD's articulation DOFs, its loop-closure joints (excluded from the articulation) and its articulation roots. Reads the file, so it shows edits saved from the GUI |
| Positions and velocities, plus driven joints' normalized positions and limits |
| Per-joint travel and mean tracking error since the last reset |
| Viewport capture, returned as an image. Needs |
| Point the viewport camera |
Drive
Tool | What it does |
| Normalized targets, as a full vector or |
| Named poses from the robot config. |
| Advance N physics steps (default dt 1/120 s) |
| Run continuously, or pause |
| Sweep every driven joint through its range and return travel stats |
| Drive every joint from its lower limit toward a target and report the fraction of travel reached. Below 0.9 counts as blocked (self-collision, a binding linkage, too little effort) |
Scene
Tool | What it does |
| Add a prop to the live scene: |
| Every prop's current world pose |
| Delete a prop |
Capture and record
These work headless, with no GUI or viewport. They use a dedicated camera that is independent of the GUI view.
Tool | What it does |
| Aim the capture camera so the whole robot fills the frame, from a given direction. |
| Place the capture camera by hand |
| A plain colored panel behind the robot. Use it with |
| One frame, returned as an image |
| Record every Nth physics step, with a caption drawn on each frame, to an animated GIF under |
Tune
Tool | What it does |
| Live |
| Live software-mimic ratios, keyed by follower joint |
| Try several values of one live parameter ( |
Train
Tool | What it does |
| Launch a headless training run in its own container and return at once. |
| Running and recent runs, and log directories holding checkpoints |
| Container state, checkpoints and latest scalars |
| Tail a running container's output |
| List TensorBoard tags, or get a downsampled series for one tag |
| Checkpoints with step and size |
| Stop a run |
Training runs do not depend on the daemon or on the MCP session. They keep going after the client disconnects.
Robot config
A robot is one JSON file. Only usd is required. Unknown keys are rejected, so a typo fails loudly instead of quietly falling back to a default.
{
"name": "franka",
"usd": "{ISAACLAB_NUCLEUS_DIR}/Robots/FrankaEmika/panda_instanceable.usd",
"fix_root_link": true,
"spawn_pos": [0, 0, 0],
"init_joint_pos": {"panda_joint4": -2.81, "panda_joint6": 3.04, ".*": 0.0},
"driven_joints": ["panda_joint[1-7]", "panda_finger_joint.*"],
"passive_joints": [],
"couplings": [{"leader": "joint_a", "follower": "joint_b", "ratio": 1.0}],
"actuator": {"stiffness": 400, "damping": 40, "effort": 87, "velocity": null},
"solver": {"pos_iters": 32, "vel_iters": 4, "self_collisions": true},
"camera": {"eye": [1.8, 1.8, 1.4], "target": [0, 0, 0.4]},
"poses": {"ready": {"panda_joint[1357]": 0.5, "panda_finger_joint.*": 1.0}}
}Key | Meaning |
| Local path (relative paths resolve from the JSON's own directory), a URL, or a path using |
| Spawn pose in the joints' own units (rad / m), keyed by regex. It must lie inside every joint's limits or spawning fails. The default is all zeros, which is out of range for e.g. Franka's joint 4 |
| Regexes (full match) for joints that take commands. Default |
| Joints whose angle is owned by a constraint, such as a closed-chain linkage. They get a zero-stiffness drive, because a live PD drive fights the constraint and the mechanism jitters |
| Software mimic joints: follower target = |
| Implicit PD gains and effort/velocity limits for the driven joints. Live-tunable |
| Spawn properties. Changing them needs |
|
|
Using your own robot without forking
Keep the robot config and assets in your own repo, and add them to the container with a compose override that you list in COMPOSE_FILE. Set it in the MCP server's environment, using absolute paths:
# my-robot/mcp-compose.yaml
services:
simd:
volumes:
- /abs/path/my-robot:/workspace/my-robotclaude mcp add isaac \
-e COMPOSE_FILE=/abs/path/dex-isaac-mcp/docker/docker-compose.yaml:/abs/path/my-robot/mcp-compose.yaml \
-e ISAAC_MCP_ROBOT=/workspace/my-robot/robot.json \
-- python -m dex_isaac_mcpTraining defaults
Out of the box, train_start runs Isaac Lab's stock skrl script inside the isaac-lab service. It tags the run name onto the log directory (logs/skrl/<experiment>/<timestamp>_ppo_torch_<run_name>/), which the other train_* tools use to find the run. To use your own launcher, set these in the environment the MCP server starts in:
Variable | Default |
| the clone this package was installed from (editable install); required for a PyPI install |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| robot config |
|
|
The script must accept --task, --headless and, when given, --num_envs, --seed, --max_iterations and --checkpoint. Tasks from your own extension need to be importable inside the container, either installed into the image or mounted.
Design notes
These are the constraints the code is built around. Most were learned by breaking them.
Every Kit call happens on the main thread. Kit, PhysX and USD are not thread-safe. Socket threads only parse JSON and queue requests, and the main loop executes them between physics steps. Answering from a reader thread appears to work, then corrupts the stage under load.
Spawn properties are frozen. Replacing a spawned articulation needs
SimulationContext.stop(), which blocks on a timeline event that only advances while the Kit loop pumps. A command runs on that loop, so the call never returns.omni.usdnew_stage()has the same trap. So the USD, solver iterations and self-collision need a restart (sim_reload), and gains stay live. Undoing an experiment does not:sim_resetwrites the spawn state back in a few frames.A Unix socket, not TCP. The repo is bind-mounted and the container runs as the host uid, so the host sees the socket file directly, with no port mapping. Paths are capped at 107 bytes (
AF_UNIX). If your checkout is deep, setISAAC_MCP_SOCKET.The host side imports no Isaac code.
protocol.py,robot.pyandtraining.pyare stdlib-only. The MCP server adds onlymcp. Nothing on the host needs isaaclab, torch or a GPU.Cache directories are committed with
.gitkeep. If Docker auto-creates a bind-mount source, it is root-owned, and Kit then dies withregistry cache path is not setbefore any script runs.The base image is pinned by digest. A re-pulled tag once shipped
/isaac-simas mode 750, and every non-root container lost its Python.Recorded GIFs are stabilized. The renderer's denoiser shimmers: between two frames of a motionless scene, about 9% of background pixels change slightly, and a GIF re-encodes every one of them. Holding sub-threshold changes and using one shared palette took a 9-second clip from 15 MB to 1.2 MB.
The daemon always renders, even headless (
enable_cameras). Without rendering, PhysX never registers a prop spawned at runtime. Prop poses are read from fabric, because the USD transform and the PhysX CPU query both stay at the spawn pose, and creating a PhysX tensor view mid-simulation crashes CUDA.No floor for range tests (
ground=False) on anything whose links can reach the ground. Otherwise the test measures the floor, not the robot.
Development
python -m unittest discover tests # host-side tests: no Isaac, no GPU
ruff check .dex_isaac_mcp/protocol.Client is a handy debugging client:
from dex_isaac_mcp.protocol import Client
with Client() as c:
print(c.call("status"))
c.call("set_pose", name="ready"); c.call("step", n=240)Status
Tested against Isaac Lab 2.3.2 (Isaac Sim 5.x) and mcp 2.3, over the stdio protocol, with the GUI on:
Franka and Allegro examples, plus a custom closed-linkage hand through a compose override:
sim_up, poses, screenshots,sim_range_test,sim_sweep(restores the original value),sim_down.Headless daemon: gains, frozen-parameter rejection, wave.
Headless capture and recording, Allegro and Franka: auto-framing, backdrop, captions, GIF output (the clip at the top).
Props, GUI and headless: a sphere and a cylinder dropped onto a static table settle at exactly table height plus their radius and half-height.
Training, against Isaac Lab's stock skrl script:
Isaac-Cartpole-v0launched, polled, logged, checkpointed and read back through everytrain_*tool, plus a run stopped mid-training. skrl'swrite_interval: autowrites no TensorBoard scalars on a very short run (5 iterations), sotrain_metricscomes back empty there; 50 iterations gives 18 tags.
Issues and PRs are welcome.
License
MIT, see LICENSE.
Available Tools
26 toolssim_downA
Stop the sim daemon; its container removes itself.
Inputs: no named parameters. Returns: object
sim_get_joint_stateB
Joint positions (rad/m) and velocities, plus driven joints' normalized positions and limits.
Inputs: no named parameters. Returns: object
sim_get_statsA
Per-joint travel and driven-joint mean tracking error since the last reset.
Travel on a passive joint proves a linkage transmits: with zero stiffness it moves only if its constraint moves it.
Inputs: no named parameters. Returns: object
sim_inspect_jointsA
List a USD's articulation DOFs, joints excluded from the articulation (loop closures), and roots.
Reads the file, not the live scene, so it shows edits saved from the Isaac GUI. Defaults to the loaded USD.
Inputs: usd. Returns: object
sim_list_posesB
Named poses from the robot config, as {pose: {joint_pattern: 0..1}}.
Inputs: no named parameters. Returns: object
sim_playA
Run the sim continuously in real time (True) or pause it (False).
Inputs: playing. Returns: object
sim_range_testA
Drive every driven joint from its lower limit toward target, report the fraction reached.
A joint below 0.9 is blocked: self-collision, a binding linkage, or too little effort for the load. reset=True first puts every joint at its lower limit and verifies it. Start the daemon with ground=False if the robot can reach the floor, or the test measures the floor.
Inputs: reset, steps, target. Returns: object
sim_reloadA
Restart the daemon to change a spawn property: robot, USD, solver iterations, self-collision.
One Kit start rather than one per experiment; gains stay live via sim_set_params. Omitted robot/usd fall back to the defaults, not to what was loaded before.
Inputs: gui, usd, robot, ground, +2 more. Returns: object
sim_screenshotA
Capture the viewport as an image. Requires the daemon started with gui=True.
Inputs: no named parameters.
sim_set_cameraB
Point the viewport camera, e.g. eye=[1.2, 1.2, 1.0] target=[0, 0, 0.3].
Inputs: eye (required), target (required). Returns: object
sim_set_couplingA
Change software-coupling ratios live, keyed by follower joint. 0 releases a follower.
Couplings are declared in the robot config; a follower is driven to ratio x its leader's target.
Inputs: ratios (required). Returns: object
sim_set_paramsC
Change the driven joints' PD gains and effort limit on the live articulation.
Inputs: effort, damping, stiffness. Returns: object
sim_set_poseA
Command a named pose. amount blends toward it: 1 = as authored, 0.5 = halfway.
Halfway from the lower limits by default, or from the current targets with from_current=True. Follow with sim_step, then sim_screenshot to verify.
Inputs: name (required), amount, from_current. Returns: object
sim_set_targetsA
Command the driven joints, normalized 0..1 (lower..upper limit).
Pass unit as a full vector in driven order (see sim_status), or joints
to set individual joints by name. Targets persist; advance with sim_step.
Inputs: unit, joints. Returns: object
sim_statusA
Report whether the sim daemon is up, its robot, driven joints, poses and parameters.
Inputs: no named parameters. Returns: object
sim_stepA
Advance the simulation n physics steps (dt from sim_status, default 1/120 s).
Inputs: n. Returns: object
sim_sweepA
Compare several values of one live parameter inside the single session.
param: stiffness, damping, effort, or "coupling:". For each value
it applies the parameter and runs test ("wave" or "range") for steps.
The original value is restored afterwards, even on error. A diverging
solver is recorded as a result, not raised.
Inputs: param (required), values (required), test, steps. Returns: object
sim_upA
Start the daemon container and wait until it answers. Idempotent.
robot: a robot config JSON, path relative to the repo root
(e.g. examples/robots/franka.json). usd: a USD path instead of / overriding
the config's (every joint driven if no config). gui=True is needed for
sim_screenshot; the host must have run xhost +local:docker once.
ground=False spawns without a floor (use for range tests).
extra_args go to scripts/simd.py verbatim (e.g. ["--pos-iters", "64"]).
The first start downloads Nucleus assets and builds shader caches: minutes.
Inputs: gui, usd, robot, ground, +2 more. Returns: object
sim_waveB
Sweep every driven joint through its range for n steps; return travel stats.
Inputs: n. Returns: object
train_checkpointsB
List a run's saved checkpoints with step and size.
Inputs: run_name (required). Returns: object
train_listA
List training runs: live or recent containers, and log directories holding checkpoints.
Inputs: no named parameters. Returns: object
train_logsA
Tail a running training container's stdout/stderr.
Inputs: run_name (required), tail. Returns: object
train_metricsA
TensorBoard scalars for a run: no tag lists tag names; a tag returns its series, downsampled.
Inputs: run_name (required), tag, max_points. Returns: object
train_startA
Launch a headless Isaac Lab training run in its own container; returns at once.
task is a registered gym id (e.g. Isaac-Cartpole-v0). run_name defaults to '_'; keep it to find the run later. extra_args go to the training script verbatim (e.g. Hydra overrides). device pins one GPU index. Independent of the daemon and of this MCP session.
Inputs: task (required), seed, device, num_envs, +4 more. Returns: object
train_statusB
Container state, checkpoints, and latest TensorBoard scalar values for one run.
Inputs: run_name (required). Returns: object
train_stopC
Stop a training run's container.
Inputs: run_name (required). Returns: object
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 26 tools
Tools are cleanly partitioned into sim_* and train_* domains, and within sim the verbs target distinct actions (status, step, play, set_targets vs set_pose vs set_coupling). Minor overlap between sim_wave, sim_range_test, and sim_sweep is resolved by their descriptions.
All tool names use snake_case with consistent sim_ or train_ prefixes and predictable verb_noun structure, e.g. sim_set_targets, sim_get_joint_state, train_start, train_checkpoints.
26 tools is slightly above the ideal single-domain range, but the server covers two substantial subsystems: live Isaac Sim control and headless Isaac Lab training orchestration. The count is reasonable for that dual scope.
The surface covers sim lifecycle (up/down/reload, status, joint state, poses, params, camera, screenshot, experiments) and training lifecycle (start/list/status/logs/stop/checkpoints/metrics). Minor gaps include no explicit generic sim reset and no training-run cleanup/delete operation.
Maintenance
Related MCP Connectors
Drive real devices from your AI Coding tool. Embed a client SDK (Unity, Godot, Flutter, iOS/macOS, Android, React Native, Web) in your app, then capture screenshots, traverse the UI tree, inject taps and key events, and run automated test tasks on the physical device over a secure relay.
Control Unreal Engine to browse assets, import content, and manage levels and sequences. Automate…
Run and manage your Aeon autonomous agent: skills, runs, memory, strategy, packs, settings.
LLM chat, text tools, image generation, editing, batch image jobs, and asynchronous video generation
Related MCP Servers
- AlicenseAqualityCmaintenanceEnables driving Omniverse Kit apps (Isaac Sim, Isaac Lab) over MCP, allowing agents to control simulations, run Python, and call namespace-scoped tools via a single bridge.13Apache 2.0
- AlicenseCqualityCmaintenanceEnables AI assistants to control NVIDIA Isaac Sim by building scenes, loading assets, operating robots and humans, reading sensors, managing simulation, and creating Action Graphs.1291MIT
- AlicenseAqualityCmaintenanceEnables MCP-capable agents to see and control robots directly by listing tasks, starting and resetting episodes, observing camera feeds and robot state, and moving end effectors and grippers, including coordinating two arms. Supports LIBERO, RoboDojo, a simulator-free fake backend and remote backends over SSH or TCP, with configurable episode budgets, logs, live monitoring and frame-by-frame replay.744 PyPI5Apache 2.0
- AlicenseAqualityCmaintenanceLets agents see and drive a live 3D application session through its addon socket: viewport screenshots, turntable and multi-mode renders, scene and object inspection, arbitrary Python execution, GLB import/export and asset validation, plus rig, pose-deformation and silhouette-likeness checks. This lets an agent visually judge a 3D result instead of guessing.25MIT