SimScene MCP Server
Provides tools to discover Hugging Face dataset repositories, list USD paths, and acquire selected USD assets at a full commit SHA with resolvable dependencies.
Provides access to NVIDIA USD Search and Isaac Sim for searching and acquiring USD assets, building editable scenes, running simulations, recording camera evidence, and verifying physical simulation runs.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@SimScene MCP Server find an industrial conveyor asset and build a scene with it"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
SimScene
Describe a scene. Find assets online. Build, simulate, and film it.
SimScene gives AI agents tools to discover simulation assets, compose editable USD scenes, run Isaac Sim, and improve the result from real images and physical observations. It connects online asset selection to scene creation and video delivery through MCP and reusable agent skills.
V2 preview · Asset catalog · 中文快速开始 · Plugin setup · Examples · Apache-2.0

Watch the V2 kinetic gallery (66s) · Watch the chain reaction · Watch the sorting workshop
What you can build
Capability | What SimScene provides |
Online asset discovery | Natural-language NVIDIA USD Search, asset inspection, previews, and Hugging Face repository discovery. |
On-demand acquisition | Fetch selected assets and dependencies into a bounded cache on the simulation host. No full asset library download is required. |
Editable composition | USD scenes, measured placement, support relationships, materials, lights, cameras, and reusable scene recipes. |
Physical interaction | Isaac Sim execution, state and contact observations, and checks against a case defined before the run. |
Simulation films | Native camera recording, editable camera plans, video editing, and frame-level source tracking. |
Agent workflows | An MCP server plus skills for building scenes and delivering simulation videos. |
The project has completed a stage covering online asset composition, physical interaction validation, and multi-camera film delivery, with reusable production tools. Measured comparisons have not established token savings across tasks; see the results and limitations.
Related MCP server: IsaacSim-MCP
Get started
Recommended first run: pin v0.2.0-dev.1 → configure the tested runtime → select a small ABO/GSO object → check a single object's fall, contact and settling → expand to multiple instances → record video. Follow the preparation setup before attempting GLB physics. The gravity-cell example documents the single-object and four-instance checks; its NVIDIA table follows the separate legacy acquisition route described below.
Your setup | What to try next |
Base SimScene installation and network access | Search metadata and inspect candidates; this does not require the GLB decoder or start Isaac Sim. |
Compatible Isaac Sim and USD bindings, pinned Node.js decoder, successful | Prepare a selected ABO/GSO GLB, then run fall/contact/settling checks. |
Verified physical scene and configured FFmpeg | Plan cameras, capture native frames, inspect and export video. |
Python 3.10+ is required. Running simulations also requires a compatible NVIDIA Isaac Sim installation on a GPU host. Install FFmpeg for video capture and editing; see video setup for encoding requirements.
git clone https://github.com/ros-claw/simscene.git
cd simscene
git checkout v0.2.0-dev.1
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -e . usd-core
simscene configure --isaac-python /absolute/path/to/isaac-sim/python.sh
simscene doctor
simscene search 'industrial conveyor'Search returns candidates and metadata. After choosing a result, inspect it and acquire only the selected files:
simscene inspect 'ASSET_URL_FROM_SEARCH'
simscene acquire 'ASSET_URL_FROM_SEARCH'The default service is NVIDIA USD Search.
Set USD_SEARCH_API_URL to use another endpoint. Check each asset's dependency
status and license before use. Downloaded models and textures are not bundled
with this repository.
V2 catalog and preparation
The V2 development line is 0.2.0.dev1 (plugin 0.2.0-dev.1). It adds unified metadata search, immutable preparation plans, persistent preparation jobs, role-based scene recipes and optional isolated Sand3D generation. See the source registry and measured validation scope.
simscene catalog sources
simscene catalog search 'notebook' --source abo-glb --limit 4
simscene recipe listSearch retrieves metadata; selecting and preparing a model is a separate step. Default sources are NVIDIA USD Search, ABO and GSO. Explicit optional providers cover a reviewed native USD manifest, a selected BOM package, configured HDRI IDs and Central metadata locators. Asset licenses, dependency completeness and physical suitability are checked individually. Source admission does not certify every model.
The example NVIDIA PackingTable's individual license remains unknown, so the
new asset_plan route blocks its preparation. Enabling NVIDIA search by default
does not authorize preparing every result. The gravity-cell example documents
its separate legacy acquisition route and makes no redistribution claim.
For imported dynamic objects, use the prepared package's actual body mapping and verify contacts, movement and settling. The gravity-cell example provides a complete selected-asset route. Preparation, scene recipes, and optional generation describe setup and reuse. Existing asset and simulation tools remain compatible.
Connect your agent
Run the MCP server from the installed environment:
simscene mcpFor an MCP client that accepts a JSON server configuration:
{
"mcpServers": {
"simscene": {
"command": "/absolute/path/to/simscene/.venv/bin/python",
"args": ["-m", "simscene.cli", "mcp"]
}
}
}Use the build-scene skill to discover assets, compose a scene, run it, and inspect the result. Use deliver-scene-video to plan cameras and produce traceable films. Packaging and installation covers portable, Claude Code, and DeepSeek Harness integration.
Try a prompt such as:
Build a small workshop with a conveyor, collection bins, and a camera that clearly shows the handoff. Find suitable assets online, inspect their scale and dependencies, run the scene, and improve it from the observations.
Examples
Example | Explore |
Only one sugar cube | Editable whole-home experiment · 55-second film · Red-rope control |
Multi-source kinetic gallery | |
Cross-catalog gravity cell | |
Three-level chain reaction | |
Precision sorting workshop | |
Maintenance workbench |
Recipes preserve the selected asset references and scene operations. Acquire the referenced assets on your own simulation host and use the example's case to check a new run.
Remote simulation
The MCP service can run on a GPU workstation while assets, USD files, and caches stay on that host. Clients receive images, state, and video data. For the experimental HTTP transport, bind through an SSH tunnel as described in the remote setup guide. A local SSH tunnel has been tested; cross-machine deployment still needs validation in the target environment.
Documentation
SimScene is a development project. Scene plans and successful worker exits need physical verification against the intended task. Robot following and collision checks are experimental; their measured scope is described in the validation documentation.
License
SimScene code is licensed under Apache-2.0. Third-party assets retain their own terms; see NOTICE.
Available Tools
52 toolsasset_acquireA
Start downloading a selected asset and dependencies. Poll job_status until completed; its result.root is the local USD path for scene_apply. Check dependency_status and unresolved_dependencies before use.
Inputs: asset_url (required), refresh.
asset_generateB
Start a durable job for an explicitly configured independent Sand3D runtime and a shipped fixed template (guide_plate_v1 or open_receiver_v1). Accepts structured dimensions only, never external programs. Completion returns a generated AssetRef for ordinary asset_plan/asset_prepare; generation alone gives no physical pass.
Inputs: recipe_id (required), parameters (required), request_id.
asset_inspectBRead-only
Retrieve indexed metadata, dependency graph and download estimate.
Inputs: asset_url (required).
asset_inspect_localBRead-only
Inspect a USD file on the simulation host. Physics behavior is not verified.
Inputs: asset_path (required).
asset_planA
Freeze a selected revision, role, toolchain and transfer budget without downloading models or starting physics. Unknown constraints block preparation.
Inputs: asset_ref (required), parameters, profile_id, budget_bytes.
asset_prepareB
Start a persistent acquisition/preparation job; poll job_status. Preparation alone does not establish runtime physics success. Reuse request_id to reconnect.
Inputs: plan_id (required), request_id.
asset_searchCRead-only
Find candidate assets online without downloading 3D assets.
Inputs: limit, query, filters, image_data_url.
asset_thumbnailCRead-only
Return a preview by search hit thumbnail_image_key or asset URL.
Inputs: offset, asset_url, image_key.
catalog_discoverBRead-only
Discover bounded HF repository candidates, not enabled libraries. No models, conversion or simulation; review access/license/layout facts before registering a source.
Inputs: query (required), budget, cursor.
catalog_searchARead-only
Search reviewed sources with bounded metadata only. Read known source IDs from simscene://catalog/sources or simscene catalog sources; copy those IDs into filters.source_ids for explicit selection. Detailed records are linked, not inlined.
Inputs: query (required), limit, cursor, filters.
environment_configureC
Save non-secret local defaults; environment variables take precedence.
Inputs: cache_bytes, isaac_python, data_directory, usd_search_url, +1 more.
environment_probeARead-only
Inspect configured runtime and GPU without starting Isaac Sim.
Inputs: no named parameters.
huggingface_acquire_assetB
Download one selected dataset USD and resolvable dependencies at a pinned commit; poll job_status.
Inputs: path (required), repo_id (required), revision (required).
huggingface_list_assetsBRead-only
List USD files from one page of a dataset at a supplied revision.
Inputs: repo_id (required), revision (required), path, limit, +1 more.
huggingface_search_datasetsBRead-only
Discover Hugging Face dataset repositories by name.
Inputs: query (required), limit.
job_cancelB
Request cancellation; poll status for confirmation.
Inputs: job_id (required).
job_statusA
Check a job; optionally wait machine-side and save a compact completion receipt. For wait_seconds>0, the caller must choose a new unused absolute receipt_path in their own output directory. A timeout preserves the same job_id; wait again with a new unused receipt path. Waiting grants no physical acceptance. Configure a client timeout longer than wait_seconds.
Inputs: job_id (required), receipt_path, wait_seconds.
robot_plan_pick_cycleCRead-only
Propose experimental robot-specific phases for an approximately 4 cm cube; use $slot and $item to expand with scene_plan_stream. Validate all paths and physical results.
Inputs: drop_position (required), model_directory (required), item, workspace_max, +2 more.
robot_prepare_profileB
Prepare experimental controller files for one acquired USD on the simulation host. Models: kawasaki_rs007l_rg2, franka_panda, ur10_short_suction. Poll job_status; validate compatibility and physical behavior before use.
Inputs: model (required), asset_path (required).
robot_solve_ikBRead-only
Find a URDF IK candidate in the base frame; does not check collisions.
Inputs: base_link (required), tool_link (required), urdf_path (required), joint_names (required), +3 more.
robot_suggest_configurationBRead-only
Suggest an experimental starting controller configuration for a generated profile and world base pose; validate actual reachability, collisions and behavior in Isaac.
Inputs: base_position (required), model_directory (required), orientation_wxyz (required).
run_contactsBRead-only
Page recorded contact_counts, robot_contacts, or optional overlap_counts from a completed run. Overlap counts include expected contacts and do not measure penetration.
Inputs: job_id (required), limit, offset, category.
run_observeBRead-only
Read the latest state while running or after failure, or a sample from a completed run.
Inputs: job_id (required), sample_index.
run_presentation_eventsARead-only
Read semantic phase/spawn events from the completed run's frozen spec. No event implies physical success; inspect run_verify separately.
Inputs: job_id (required).
run_verifyARead-only
Check completed physical evidence against an independent pinned local case. Types: contact, suction, moving, repeated, domino, parallel, stream, mechanism, passive_network. Passive schemas infer their type. Mechanism detached_drawer and network no_trigger/broken_rope/locked_door modes require the matching reference_job_id and isolated declared changes. A single positive network run does not accept the whole stage: use run_verify_passive_suite with its frozen required controls. This verifier only reads measurements and never releases or moves a mechanism.
Inputs: job_id (required), mode, case_path, case_type, +1 more.
run_verify_evidenceBRead-only
Recheck saved output hashes; this is not a signature or task-success check.
Inputs: job_id (required).
run_verify_lockBRead-only
Recheck saved run inputs and external dependencies against recorded hashes.
Inputs: job_id (required).
run_verify_passive_suiteARead-only
Read-only acceptance of a frozen passive-network positive plus its required isolated controls. control_job_ids maps no_trigger, broken_rope and/or locked_door to completed distinct job IDs. Case bytes, source changes, sampled physical clocks, runtime settings and evidence locks are checked; missing controls cannot accept a P2/P3 stage. This checks physical acceptance, not final video or artistic quality.
Inputs: control_job_ids (required), positive_job_id (required), case_path.
run_video_chunkCRead-only
Read a bounded, base64-encoded chunk of a completed camera recording.
Inputs: camera (required), job_id (required), length, offset.
scene_applyB
Atomically edit a revision. For a prepared asset, use asset_path=result.root and asset_binding={"manifest_path":result.manifest_path,"manifest_sha256":result.manifest_sha256} with exactly these two binding keys from the preparation result. A rigid_prop_v1 asset requires dynamic=true. Set rotation with normalized orientation_wxyz=[w,x,y,z]. Supported kinds: box, sphere, cylinder, capsule, compound, joint, asset, camera, light, counter; make ground from a static box. A light can use light={"type":"dome"}, {"type":"sphere","radius_m":0.2}, or {"type":"rect","width_m":2,"height_m":1}; default is distant. Configure intensity, optional light color/exposure and transform. A counter uses counter={"source":"spawned"} or {"source":"verified_archives","class_label":"red"}; verified counts require case-checked archiving. A non-colliding static box can use indicator={"robot_id":"arm"} to show the observed controller phase. A pose_follower can opt into reactive obstacle planning with avoid_obstacles listing static collision-enabled primitive IDs and obstacle_margin_m; this is not collision safety certification. A USD asset requires the local simulation-host asset_path returned by acquisition, not its online URL. Example: {"op":"add","id":"box1","object":{"kind":"box","size":[0.2,0.2,0.2],"position":[0,0,0.1]}}. IDs belong on the operation; object uses kind. Update uses {"op":"update","id":"box1","changes":{...}}. Also supports remove, place_on, arrange_line, arrange_grid, align and look_at. compound uses 1..128 local primitive parts under one unscaled rigid body; part visible=false hides its rendered proxy while retaining collision. An acquired asset with asset_physics=visual_only and asset_lights=disabled can set rigid_attachment={body:compound_ID}; its position/orientation then define a local pose under that dynamic compound. It follows USD hierarchy, with no runtime pose commands. Measure the original mesh and disclose its collision proxy approximation. joint={type:revolute|prismatic|spherical|fixed|d6,body0:null|ID,body1:ID,axis:X|Y|Z,local_pos0:[...],local_pos1:[...],lower_limit:...,upper_limit:...}; angular limits are degrees, linear limits metres. D6 instead requires dof_limits with all6 transX/transY/transZ/rotX/rotY/rotZ values: free, locked, or [low,high] with low<high; no axis/lower_limit fields. Joint X is twist; local_rot0_wxyz/local_rot1_wxyz define frames. D6 swing bounds form a pyramid, not a cone; use measured relative frame quaternion rather than global Euler components. Primitive kind=joint remains passive, with no drive targets. A prepared articulated_prop_v1 asset can use mechanism={"commands":[{"joint_id":"main","trigger":{"type":"at_step","physics_step":0},"target_position":0.5,"position_unit":"rad","stiffness":400,"damping":12,"max_effort":22}]}; use manifest joint IDs and limits, rad for hinges or m for sliders. Commands declare a schedule; actual joint response is observed separately. Initial body poses define initial state. Nonuniform legacy primitives need a baked compound frame when jointed. set_observation with observation={sample_interval_steps:1,strict_start:true,record_contact_pairs:true} requests every actual physics step, a measured reset zero after SDK warmup, and actual per-step contact records; no body setters run during the task. Imported supports need support_surfaces={name:{min_xy:[x,y],max_xy:[x,y],z_m:z,exclude:[]}} in normalized local metres; place_on uses support_surface and offset_xy with optional avoid_ids/keepout_bounds/clearance_m. arrange_grid uses ids,support,support_surface,columns,gap_m; align uses ids,axis,edge,value_m. set_presentation accepts presentation.profile=technical_showcase_1080p and capture/output/budget/diagnostics; camera settings live under camera={resolution:[width,height],focal_length_mm:35,horizontal_aperture_mm:36,...}. Native 30fps records every two physics steps. For a static box surface use {"op":"bind_conveyor","id":"belt","speed_m_s":0.1,"direction":[1,0,0]}; imported assets cannot be conveyor surfaces directly. A camera can use camera_orbit={"center":[0,0,0.5],"radius_m":2,"height_m":1,"duration_s":8} for a circular view.
Inputs: scene_id (required), operations (required), request_id (required), base_revision (required).
scene_compile_recipeARead-only
Compile a versioned recipe into ordinary scene_apply operations at the current base_revision. Supply role bindings with asset_path and pinned asset_binding. This does not edit or run the scene and gives no physical pass; apply the full batch atomically, then freeze and verify a new trial.
Inputs: bindings (required), scene_id (required), recipe_id (required), parameters (required).
scene_exportC
Export supported scene objects to a USD file on this server.
Inputs: scene_id (required), destination (required).
scene_getCRead-only
Get the current scene specification.
Inputs: scene_id (required).
scene_measureARead-only
Measure authored world render bounds and bottom anchors, including imported units/axes/variants and rigid visual attachments. Optional include_overlap_candidates returns AABB intersections among selected objects to review initial layout interference; empty compound space and deliberate contact can be false positives. It does not certify physical clearance. Imported outer bbox is not a support surface.
Inputs: ids (required), scene_id (required), include_overlap_candidates.
scene_observeCRead-only
Return a live/last preview (when enabled) or a completed camera image.
Inputs: camera (required), job_id (required).
scene_openC
Create or reopen a scene specification.
Inputs: scene_id (required).
scene_plan_collision_auditCRead-only
Propose discrete overlap monitoring and model-specific mechanical exceptions for supported robot paths or matching generated profiles. Include every robot in the stream case, review exclusions, apply operations, regenerate the case, add collision_policy and run verification. This does not certify swept-path or real-robot safety.
Inputs: scene_id (required), robot_ids (required).
scene_plan_interleaveCRead-only
Propose a class-aware round-robin spawn schedule while preserving each class's item order. Review robot timing and other objects, apply operations, regenerate the case, and rerun physics.
Inputs: ids (required), scene_id (required), class_order (required), interval_steps (required), +1 more.
scene_plan_presentationBRead-only
Propose measured overview/detail cameras and a capture profile. regions use id,min,max, optional action/subject_ids. Review occlusion, lighting and existing camera budget; apply operations and preview before native recording.
Inputs: scene_id (required), subject_ids (required), profile, regions, +3 more.
scene_plan_retimeBRead-only
Propose retiming selected scheduled objects, preserving their order and first spawn. Review and apply the operations, then rerun.
Inputs: ids (required), scene_id (required), spacing_factor (required).
scene_plan_ropeBRead-only
Propose real colliding capsule links connected by passive spherical joints. points are a metric world polyline, including deliberate slack. Optional start/end attachments each use body and local_position_m. Apply returned ordinary operations after review. This is not a distance-only or animated rope; routing over a free pulley and actual tension transfer need physical trials and broken-end controls.
Inputs: points (required), prefix (required), scene_id (required), physics, +5 more.
scene_plan_streamARead-only
Propose scheduled workpieces and per-class robot cycles without editing the scene. Each routes[label] uses robot_id, wait={axis,minimum,maximum,max_steps}, cycle=[phases], and optional slots. Use the cycle returned by robot_plan_pick_cycle; apply returned operations with base_revision.
Inputs: size (required), labels (required), prefix (required), routes (required), +6 more.
scene_plan_stream_caseCRead-only
Propose a scene-derived physical evidence case using explicit per-class inner bin bounds and forbidden objects. This is not independent acceptance or path-safety certification.
Inputs: bins (required), belt_id (required), scene_id (required), forbidden_objects (required).
scene_plan_swap_routesBRead-only
Propose swapping two equal-sized, single-class scheduled robot routes. Review bin labels, reachability and physical results.
Inputs: scene_id (required), first_robot (required), second_robot (required).
scene_plan_synchronizeCRead-only
Propose interleaved spawn times from observed pickup delays in an intact completed stream run. Review and apply the proposal, regenerate the case, and verify simultaneous cycles and safety in a new run; this is not a guarantee.
Inputs: ids (required), job_id (required), scene_id (required).
scene_plan_translate_groupARead-only
Propose moving objects by a world-space delta and optionally moving one robot's explicitly selected nearby absolute waypoints. Review the returned phase indices and rerun.
Inputs: ids (required), delta (required), scene_id (required), move_targets.
scene_validateCRead-only
Check supported object schemas. This is not physics validation.
Inputs: scene_id (required).
simulation_pauseB
Request a cooperative timeline pause; poll job_status for acknowledgment.
Inputs: job_id (required).
simulation_resumeA
Request resumption of a paused simulation; poll job_status.
Inputs: job_id (required).
simulation_startB
Start or queue an experimental Isaac job, optionally pinning an inline case or server-side JSON/YAML case. Reusing the same request_id with identical parameters returns the original job; use a new ID for a distinct run. continue_on_controller_error keeps physics running after a controller failure. archive_verified_objects requires a pinned stream case.
Inputs: scene_id (required), case, seed, steps, +4 more.
video_exportB
Edit intact real run recordings. Segments require run_directory,camera,start_sim_time_s,end_sim_time_s; optional speed,title. Preserve native cadence, save source hashes and sim/output time mapping. Upscaling requires explicit opt-in.
Inputs: segments (required), destination (required), output, font_file, +2 more.
video_inspectB
Probe/decode real video; when job_id/camera supplied, bind to intact source and verify native capture dimensions and physics frame clock. Representative PNG paths are returned for actual visual review.
Inputs: path (required), camera, job_id, output_dir, +1 more.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 52 tools
Most tools have distinct roles, but several clusters overlap: scene_plan_synchronize/interleave/retime/stream, asset_search/catalog_search/catalog_discover/huggingface_search_datasets, and run_verify/run_verify_lock/run_verify_evidence. Detailed descriptions help, but an agent must read carefully to avoid misselection.
Almost all names are snake_case with domain prefixes such as scene_, asset_, run_, job_, robot_, and video_, mostly following context_verb or context_verb_noun patterns. Minor deviations include huggingface_* tools breaking the asset_ prefix and job_status/simulation_start using noun-first ordering.
52 tools is far above the recommended 3-15 range and near the 50+ extreme; the domain is broad, but the surface is very heavy and increases selection load. Many specialized tools earn their place, yet the total count is still excessive for reliable agent use.
Core workflows—scene editing, asset acquisition, robot planning, simulation control, run verification, and video export—are well covered, with multiple specialized verification tools. Gaps include no explicit scene/run/job listing or deletion tools and no catalog registration tool despite discovery mentioning registration.
Maintenance
Related MCP Connectors
Generate 3D models, worlds, PBR materials, and coordinated asset packs from AI agents.
Build and run visual creative-production workflows from your AI agent.
Build, validate, and deploy multi-agent AI solutions from any AI environment.
Generate and edit images, video, voice, lip-sync and 3D models from your AI agent.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceEnables AI assistants to read, inspect, and manipulate OpenUSD 3D scenes, including scene graph traversal, property querying, mesh export, and variant switching.10MIT
- AlicenseCqualityCmaintenanceEnables AI assistants to control NVIDIA Isaac Sim by building scenes, loading assets, operating robots and humans, reading sensors, managing simulation, and creating Action Graphs.1291MIT
- AlicenseNot gradedqualityCmaintenanceEnables LLM agents to generate and validate multi-granularity storyboard assets from text prompts, including 2D SVG motion, 3D Blender depth/blockout, and SDXL ControlNet final images. It also supports vision-based self-correction and a SQLite knowledge database of successful recipes.MIT
- AlicenseNot gradedqualityCmaintenanceEnables AI agents to build models, import assets, animate scenes, render images, and save versioned projects through validated MCP tools backed by headless Blender, with a live browser viewer for inspection.GPL 3.0