Skip to main content
Glama
SimScene MCP Server

SimScene MCP Server

by ros-claw

SimScene

Describe a scene. Find assets online. Build, simulate, and film it.

SimScene gives AI agents tools to discover simulation assets, compose editable USD scenes, run Isaac Sim, and improve the result from real images and physical observations. It connects online asset selection to scene creation and video delivery through MCP and reusable agent skills.

V2 preview · Asset catalog · 中文快速开始 · Plugin setup · Examples · Apache-2.0

The multi-source kinetic gallery rendered in Isaac Sim

Watch the V2 kinetic gallery (66s) · Watch the chain reaction · Watch the sorting workshop

What you can build

Capability

What SimScene provides

Online asset discovery

Natural-language NVIDIA USD Search, asset inspection, previews, and Hugging Face repository discovery.

On-demand acquisition

Fetch selected assets and dependencies into a bounded cache on the simulation host. No full asset library download is required.

Editable composition

USD scenes, measured placement, support relationships, materials, lights, cameras, and reusable scene recipes.

Physical interaction

Isaac Sim execution, state and contact observations, and checks against a case defined before the run.

Simulation films

Native camera recording, editable camera plans, video editing, and frame-level source tracking.

Agent workflows

An MCP server plus skills for building scenes and delivering simulation videos.

The project has completed a stage covering online asset composition, physical interaction validation, and multi-camera film delivery, with reusable production tools. Measured comparisons have not established token savings across tasks; see the results and limitations.

Related MCP server: IsaacSim-MCP

Get started

Recommended first run: pin v0.2.0-dev.1 → configure the tested runtime → select a small ABO/GSO object → check a single object's fall, contact and settling → expand to multiple instances → record video. Follow the preparation setup before attempting GLB physics. The gravity-cell example documents the single-object and four-instance checks; its NVIDIA table follows the separate legacy acquisition route described below.

Your setup

What to try next

Base SimScene installation and network access

Search metadata and inspect candidates; this does not require the GLB decoder or start Isaac Sim.

Compatible Isaac Sim and USD bindings, pinned Node.js decoder, successful asset probe

Prepare a selected ABO/GSO GLB, then run fall/contact/settling checks.

Verified physical scene and configured FFmpeg

Plan cameras, capture native frames, inspect and export video.

Python 3.10+ is required. Running simulations also requires a compatible NVIDIA Isaac Sim installation on a GPU host. Install FFmpeg for video capture and editing; see video setup for encoding requirements.

git clone https://github.com/ros-claw/simscene.git
cd simscene
git checkout v0.2.0-dev.1
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -e . usd-core

simscene configure --isaac-python /absolute/path/to/isaac-sim/python.sh
simscene doctor
simscene search 'industrial conveyor'

Search returns candidates and metadata. After choosing a result, inspect it and acquire only the selected files:

simscene inspect 'ASSET_URL_FROM_SEARCH'
simscene acquire 'ASSET_URL_FROM_SEARCH'

The default service is NVIDIA USD Search. Set USD_SEARCH_API_URL to use another endpoint. Check each asset's dependency status and license before use. Downloaded models and textures are not bundled with this repository.

V2 catalog and preparation

The V2 development line is 0.2.0.dev1 (plugin 0.2.0-dev.1). It adds unified metadata search, immutable preparation plans, persistent preparation jobs, role-based scene recipes and optional isolated Sand3D generation. See the source registry and measured validation scope.

simscene catalog sources
simscene catalog search 'notebook' --source abo-glb --limit 4
simscene recipe list

Search retrieves metadata; selecting and preparing a model is a separate step. Default sources are NVIDIA USD Search, ABO and GSO. Explicit optional providers cover a reviewed native USD manifest, a selected BOM package, configured HDRI IDs and Central metadata locators. Asset licenses, dependency completeness and physical suitability are checked individually. Source admission does not certify every model.

The example NVIDIA PackingTable's individual license remains unknown, so the new asset_plan route blocks its preparation. Enabling NVIDIA search by default does not authorize preparing every result. The gravity-cell example documents its separate legacy acquisition route and makes no redistribution claim.

For imported dynamic objects, use the prepared package's actual body mapping and verify contacts, movement and settling. The gravity-cell example provides a complete selected-asset route. Preparation, scene recipes, and optional generation describe setup and reuse. Existing asset and simulation tools remain compatible.

Connect your agent

Run the MCP server from the installed environment:

simscene mcp

For an MCP client that accepts a JSON server configuration:

{
  "mcpServers": {
    "simscene": {
      "command": "/absolute/path/to/simscene/.venv/bin/python",
      "args": ["-m", "simscene.cli", "mcp"]
    }
  }
}

Use the build-scene skill to discover assets, compose a scene, run it, and inspect the result. Use deliver-scene-video to plan cameras and produce traceable films. Packaging and installation covers portable, Claude Code, and DeepSeek Harness integration.

Try a prompt such as:

Build a small workshop with a conveyor, collection bins, and a camera that clearly shows the handoff. Find suitable assets online, inspect their scale and dependencies, run the scene, and improve it from the observations.

Examples

Example

Explore

Only one sugar cube

Editable whole-home experiment · 55-second film · Red-rope control

Multi-source kinetic gallery

Editable recipe · 66-second film

Cross-catalog gravity cell

Locked assets, physical case and reproduction

Three-level chain reaction

Editable recipe and reproduction steps · Film

Precision sorting workshop

Scene recipe · Film

Maintenance workbench

Scene recipe · Film

Recipes preserve the selected asset references and scene operations. Acquire the referenced assets on your own simulation host and use the example's case to check a new run.

Remote simulation

The MCP service can run on a GPU workstation while assets, USD files, and caches stay on that host. Clients receive images, state, and video data. For the experimental HTTP transport, bind through an SSH tunnel as described in the remote setup guide. A local SSH tunnel has been tested; cross-machine deployment still needs validation in the target environment.

Documentation

SimScene is a development project. Scene plans and successful worker exits need physical verification against the intended task. Robot following and collision checks are experimental; their measured scope is described in the validation documentation.

License

SimScene code is licensed under Apache-2.0. Third-party assets retain their own terms; see NOTICE.

Available Tools

52 tools
asset_acquireA

Start downloading a selected asset and dependencies. Poll job_status until completed; its result.root is the local USD path for scene_apply. Check dependency_status and unresolved_dependencies before use.

Inputs: asset_url (required), refresh.

asset_generateB

Start a durable job for an explicitly configured independent Sand3D runtime and a shipped fixed template (guide_plate_v1 or open_receiver_v1). Accepts structured dimensions only, never external programs. Completion returns a generated AssetRef for ordinary asset_plan/asset_prepare; generation alone gives no physical pass.

Inputs: recipe_id (required), parameters (required), request_id.

asset_inspectB
Read-only

Retrieve indexed metadata, dependency graph and download estimate.

Inputs: asset_url (required).

asset_inspect_localB
Read-only

Inspect a USD file on the simulation host. Physics behavior is not verified.

Inputs: asset_path (required).

asset_planA

Freeze a selected revision, role, toolchain and transfer budget without downloading models or starting physics. Unknown constraints block preparation.

Inputs: asset_ref (required), parameters, profile_id, budget_bytes.

asset_prepareB

Start a persistent acquisition/preparation job; poll job_status. Preparation alone does not establish runtime physics success. Reuse request_id to reconnect.

Inputs: plan_id (required), request_id.

asset_thumbnailC
Read-only

Return a preview by search hit thumbnail_image_key or asset URL.

Inputs: offset, asset_url, image_key.

catalog_discoverB
Read-only

Discover bounded HF repository candidates, not enabled libraries. No models, conversion or simulation; review access/license/layout facts before registering a source.

Inputs: query (required), budget, cursor.

environment_configureC

Save non-secret local defaults; environment variables take precedence.

Inputs: cache_bytes, isaac_python, data_directory, usd_search_url, +1 more.

environment_probeA
Read-only

Inspect configured runtime and GPU without starting Isaac Sim.

Inputs: no named parameters.

huggingface_acquire_assetB

Download one selected dataset USD and resolvable dependencies at a pinned commit; poll job_status.

Inputs: path (required), repo_id (required), revision (required).

huggingface_list_assetsB
Read-only

List USD files from one page of a dataset at a supplied revision.

Inputs: repo_id (required), revision (required), path, limit, +1 more.

huggingface_search_datasetsB
Read-only

Discover Hugging Face dataset repositories by name.

Inputs: query (required), limit.

job_cancelB

Request cancellation; poll status for confirmation.

Inputs: job_id (required).

job_statusA

Check a job; optionally wait machine-side and save a compact completion receipt. For wait_seconds>0, the caller must choose a new unused absolute receipt_path in their own output directory. A timeout preserves the same job_id; wait again with a new unused receipt path. Waiting grants no physical acceptance. Configure a client timeout longer than wait_seconds.

Inputs: job_id (required), receipt_path, wait_seconds.

robot_plan_pick_cycleC
Read-only

Propose experimental robot-specific phases for an approximately 4 cm cube; use $slot and $item to expand with scene_plan_stream. Validate all paths and physical results.

Inputs: drop_position (required), model_directory (required), item, workspace_max, +2 more.

robot_prepare_profileB

Prepare experimental controller files for one acquired USD on the simulation host. Models: kawasaki_rs007l_rg2, franka_panda, ur10_short_suction. Poll job_status; validate compatibility and physical behavior before use.

Inputs: model (required), asset_path (required).

robot_solve_ikB
Read-only

Find a URDF IK candidate in the base frame; does not check collisions.

Inputs: base_link (required), tool_link (required), urdf_path (required), joint_names (required), +3 more.

robot_suggest_configurationB
Read-only

Suggest an experimental starting controller configuration for a generated profile and world base pose; validate actual reachability, collisions and behavior in Isaac.

Inputs: base_position (required), model_directory (required), orientation_wxyz (required).

run_contactsB
Read-only

Page recorded contact_counts, robot_contacts, or optional overlap_counts from a completed run. Overlap counts include expected contacts and do not measure penetration.

Inputs: job_id (required), limit, offset, category.

run_observeB
Read-only

Read the latest state while running or after failure, or a sample from a completed run.

Inputs: job_id (required), sample_index.

run_presentation_eventsA
Read-only

Read semantic phase/spawn events from the completed run's frozen spec. No event implies physical success; inspect run_verify separately.

Inputs: job_id (required).

run_verifyA
Read-only

Check completed physical evidence against an independent pinned local case. Types: contact, suction, moving, repeated, domino, parallel, stream, mechanism, passive_network. Passive schemas infer their type. Mechanism detached_drawer and network no_trigger/broken_rope/locked_door modes require the matching reference_job_id and isolated declared changes. A single positive network run does not accept the whole stage: use run_verify_passive_suite with its frozen required controls. This verifier only reads measurements and never releases or moves a mechanism.

Inputs: job_id (required), mode, case_path, case_type, +1 more.

run_verify_evidenceB
Read-only

Recheck saved output hashes; this is not a signature or task-success check.

Inputs: job_id (required).

run_verify_lockB
Read-only

Recheck saved run inputs and external dependencies against recorded hashes.

Inputs: job_id (required).

run_verify_passive_suiteA
Read-only

Read-only acceptance of a frozen passive-network positive plus its required isolated controls. control_job_ids maps no_trigger, broken_rope and/or locked_door to completed distinct job IDs. Case bytes, source changes, sampled physical clocks, runtime settings and evidence locks are checked; missing controls cannot accept a P2/P3 stage. This checks physical acceptance, not final video or artistic quality.

Inputs: control_job_ids (required), positive_job_id (required), case_path.

run_video_chunkC
Read-only

Read a bounded, base64-encoded chunk of a completed camera recording.

Inputs: camera (required), job_id (required), length, offset.

scene_applyB

Atomically edit a revision. For a prepared asset, use asset_path=result.root and asset_binding={"manifest_path":result.manifest_path,"manifest_sha256":result.manifest_sha256} with exactly these two binding keys from the preparation result. A rigid_prop_v1 asset requires dynamic=true. Set rotation with normalized orientation_wxyz=[w,x,y,z]. Supported kinds: box, sphere, cylinder, capsule, compound, joint, asset, camera, light, counter; make ground from a static box. A light can use light={"type":"dome"}, {"type":"sphere","radius_m":0.2}, or {"type":"rect","width_m":2,"height_m":1}; default is distant. Configure intensity, optional light color/exposure and transform. A counter uses counter={"source":"spawned"} or {"source":"verified_archives","class_label":"red"}; verified counts require case-checked archiving. A non-colliding static box can use indicator={"robot_id":"arm"} to show the observed controller phase. A pose_follower can opt into reactive obstacle planning with avoid_obstacles listing static collision-enabled primitive IDs and obstacle_margin_m; this is not collision safety certification. A USD asset requires the local simulation-host asset_path returned by acquisition, not its online URL. Example: {"op":"add","id":"box1","object":{"kind":"box","size":[0.2,0.2,0.2],"position":[0,0,0.1]}}. IDs belong on the operation; object uses kind. Update uses {"op":"update","id":"box1","changes":{...}}. Also supports remove, place_on, arrange_line, arrange_grid, align and look_at. compound uses 1..128 local primitive parts under one unscaled rigid body; part visible=false hides its rendered proxy while retaining collision. An acquired asset with asset_physics=visual_only and asset_lights=disabled can set rigid_attachment={body:compound_ID}; its position/orientation then define a local pose under that dynamic compound. It follows USD hierarchy, with no runtime pose commands. Measure the original mesh and disclose its collision proxy approximation. joint={type:revolute|prismatic|spherical|fixed|d6,body0:null|ID,body1:ID,axis:X|Y|Z,local_pos0:[...],local_pos1:[...],lower_limit:...,upper_limit:...}; angular limits are degrees, linear limits metres. D6 instead requires dof_limits with all6 transX/transY/transZ/rotX/rotY/rotZ values: free, locked, or [low,high] with low<high; no axis/lower_limit fields. Joint X is twist; local_rot0_wxyz/local_rot1_wxyz define frames. D6 swing bounds form a pyramid, not a cone; use measured relative frame quaternion rather than global Euler components. Primitive kind=joint remains passive, with no drive targets. A prepared articulated_prop_v1 asset can use mechanism={"commands":[{"joint_id":"main","trigger":{"type":"at_step","physics_step":0},"target_position":0.5,"position_unit":"rad","stiffness":400,"damping":12,"max_effort":22}]}; use manifest joint IDs and limits, rad for hinges or m for sliders. Commands declare a schedule; actual joint response is observed separately. Initial body poses define initial state. Nonuniform legacy primitives need a baked compound frame when jointed. set_observation with observation={sample_interval_steps:1,strict_start:true,record_contact_pairs:true} requests every actual physics step, a measured reset zero after SDK warmup, and actual per-step contact records; no body setters run during the task. Imported supports need support_surfaces={name:{min_xy:[x,y],max_xy:[x,y],z_m:z,exclude:[]}} in normalized local metres; place_on uses support_surface and offset_xy with optional avoid_ids/keepout_bounds/clearance_m. arrange_grid uses ids,support,support_surface,columns,gap_m; align uses ids,axis,edge,value_m. set_presentation accepts presentation.profile=technical_showcase_1080p and capture/output/budget/diagnostics; camera settings live under camera={resolution:[width,height],focal_length_mm:35,horizontal_aperture_mm:36,...}. Native 30fps records every two physics steps. For a static box surface use {"op":"bind_conveyor","id":"belt","speed_m_s":0.1,"direction":[1,0,0]}; imported assets cannot be conveyor surfaces directly. A camera can use camera_orbit={"center":[0,0,0.5],"radius_m":2,"height_m":1,"duration_s":8} for a circular view.

Inputs: scene_id (required), operations (required), request_id (required), base_revision (required).

scene_compile_recipeA
Read-only

Compile a versioned recipe into ordinary scene_apply operations at the current base_revision. Supply role bindings with asset_path and pinned asset_binding. This does not edit or run the scene and gives no physical pass; apply the full batch atomically, then freeze and verify a new trial.

Inputs: bindings (required), scene_id (required), recipe_id (required), parameters (required).

scene_exportC

Export supported scene objects to a USD file on this server.

Inputs: scene_id (required), destination (required).

scene_getC
Read-only

Get the current scene specification.

Inputs: scene_id (required).

scene_measureA
Read-only

Measure authored world render bounds and bottom anchors, including imported units/axes/variants and rigid visual attachments. Optional include_overlap_candidates returns AABB intersections among selected objects to review initial layout interference; empty compound space and deliberate contact can be false positives. It does not certify physical clearance. Imported outer bbox is not a support surface.

Inputs: ids (required), scene_id (required), include_overlap_candidates.

scene_observeC
Read-only

Return a live/last preview (when enabled) or a completed camera image.

Inputs: camera (required), job_id (required).

scene_openC

Create or reopen a scene specification.

Inputs: scene_id (required).

scene_plan_collision_auditC
Read-only

Propose discrete overlap monitoring and model-specific mechanical exceptions for supported robot paths or matching generated profiles. Include every robot in the stream case, review exclusions, apply operations, regenerate the case, add collision_policy and run verification. This does not certify swept-path or real-robot safety.

Inputs: scene_id (required), robot_ids (required).

scene_plan_interleaveC
Read-only

Propose a class-aware round-robin spawn schedule while preserving each class's item order. Review robot timing and other objects, apply operations, regenerate the case, and rerun physics.

Inputs: ids (required), scene_id (required), class_order (required), interval_steps (required), +1 more.

scene_plan_presentationB
Read-only

Propose measured overview/detail cameras and a capture profile. regions use id,min,max, optional action/subject_ids. Review occlusion, lighting and existing camera budget; apply operations and preview before native recording.

Inputs: scene_id (required), subject_ids (required), profile, regions, +3 more.

scene_plan_retimeB
Read-only

Propose retiming selected scheduled objects, preserving their order and first spawn. Review and apply the operations, then rerun.

Inputs: ids (required), scene_id (required), spacing_factor (required).

scene_plan_ropeB
Read-only

Propose real colliding capsule links connected by passive spherical joints. points are a metric world polyline, including deliberate slack. Optional start/end attachments each use body and local_position_m. Apply returned ordinary operations after review. This is not a distance-only or animated rope; routing over a free pulley and actual tension transfer need physical trials and broken-end controls.

Inputs: points (required), prefix (required), scene_id (required), physics, +5 more.

scene_plan_streamA
Read-only

Propose scheduled workpieces and per-class robot cycles without editing the scene. Each routes[label] uses robot_id, wait={axis,minimum,maximum,max_steps}, cycle=[phases], and optional slots. Use the cycle returned by robot_plan_pick_cycle; apply returned operations with base_revision.

Inputs: size (required), labels (required), prefix (required), routes (required), +6 more.

scene_plan_stream_caseC
Read-only

Propose a scene-derived physical evidence case using explicit per-class inner bin bounds and forbidden objects. This is not independent acceptance or path-safety certification.

Inputs: bins (required), belt_id (required), scene_id (required), forbidden_objects (required).

scene_plan_swap_routesB
Read-only

Propose swapping two equal-sized, single-class scheduled robot routes. Review bin labels, reachability and physical results.

Inputs: scene_id (required), first_robot (required), second_robot (required).

scene_plan_synchronizeC
Read-only

Propose interleaved spawn times from observed pickup delays in an intact completed stream run. Review and apply the proposal, regenerate the case, and verify simultaneous cycles and safety in a new run; this is not a guarantee.

Inputs: ids (required), job_id (required), scene_id (required).

scene_plan_translate_groupA
Read-only

Propose moving objects by a world-space delta and optionally moving one robot's explicitly selected nearby absolute waypoints. Review the returned phase indices and rerun.

Inputs: ids (required), delta (required), scene_id (required), move_targets.

scene_validateC
Read-only

Check supported object schemas. This is not physics validation.

Inputs: scene_id (required).

simulation_pauseB

Request a cooperative timeline pause; poll job_status for acknowledgment.

Inputs: job_id (required).

simulation_resumeA

Request resumption of a paused simulation; poll job_status.

Inputs: job_id (required).

simulation_startB

Start or queue an experimental Isaac job, optionally pinning an inline case or server-side JSON/YAML case. Reusing the same request_id with identical parameters returns the original job; use a new ID for a distinct run. continue_on_controller_error keeps physics running after a controller failure. archive_verified_objects requires a pinned stream case.

Inputs: scene_id (required), case, seed, steps, +4 more.

video_exportB

Edit intact real run recordings. Segments require run_directory,camera,start_sim_time_s,end_sim_time_s; optional speed,title. Preserve native cadence, save source hashes and sim/output time mapping. Upscaling requires explicit opt-in.

Inputs: segments (required), destination (required), output, font_file, +2 more.

video_inspectB

Probe/decode real video; when job_id/camera supplied, bind to intact source and verify native capture dimensions and physics frame clock. Representative PNG paths are returned for actual visual review.

Inputs: path (required), camera, job_id, output_dir, +1 more.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    C2.9/5.0

    Scored across 52 tools

    Disambiguation3/5

    Most tools have distinct roles, but several clusters overlap: scene_plan_synchronize/interleave/retime/stream, asset_search/catalog_search/catalog_discover/huggingface_search_datasets, and run_verify/run_verify_lock/run_verify_evidence. Detailed descriptions help, but an agent must read carefully to avoid misselection.

    Naming Consistency4/5

    Almost all names are snake_case with domain prefixes such as scene_, asset_, run_, job_, robot_, and video_, mostly following context_verb or context_verb_noun patterns. Minor deviations include huggingface_* tools breaking the asset_ prefix and job_status/simulation_start using noun-first ordering.

    Tool Count2/5

    52 tools is far above the recommended 3-15 range and near the 50+ extreme; the domain is broad, but the surface is very heavy and increases selection load. Many specialized tools earn their place, yet the total count is still excessive for reliable agent use.

    Completeness4/5

    Core workflows—scene editing, asset acquisition, robot planning, simulation control, run verification, and video export—are well covered, with multiple specialized verification tools. Gaps include no explicit scene/run/job listing or deletion tools and no catalog registration tool despite discovery mentioning registration.

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      D
      maintenance
      Enables AI assistants to read, inspect, and manipulate OpenUSD 3D scenes, including scene graph traversal, property querying, mesh export, and variant switching.
      10
      MIT
    • A
      license
      C
      quality
      C
      maintenance
      Enables AI assistants to control NVIDIA Isaac Sim by building scenes, loading assets, operating robots and humans, reading sensors, managing simulation, and creating Action Graphs.
      129
      1
      MIT
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables LLM agents to generate and validate multi-granularity storyboard assets from text prompts, including 2D SVG motion, 3D Blender depth/blockout, and SDXL ControlNet final images. It also supports vision-based self-correction and a SQLite knowledge database of successful recipes.
      MIT