agentic-sim-world-gen
Searches and retrieves SimReady Central 3D asset datasets hosted on Hugging Face, such as NVIDIA and ForgeCAD SimReady assets, to select and lay out assets for generated scenes. It can also optionally download Hugging Face-hosted models like SigLIP 2 and MoGe-2 for asset taste checks and photo depth estimation.
Integrates with NVIDIA Isaac Sim and NVIDIA-provided SimReady assets to build USD worlds, run physics checks, capture screenshots, use bundled Isaac Sim assets, and generate synthetic data with Replicator.
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
agentic-sim-world-gen
English | 日本語 | Project page
An agent (Claude Code) builds an Isaac Sim world (USD) from a description in Japanese or English, a floor plan or photos: it searches for assets, picks them, lays them out, checks the result with physics in Isaac Sim and records the license of every asset. Assets come from SimReady Central, the Isaac Sim bundled assets, Poly Haven CC0 models and your own USD folders.
flowchart TB
IN["<b>Input</b><br/>text (Japanese or English)<br/>floor plan: JSON / DXF / IFC / image<br/>photos / video"]
CC["<b>Claude Code</b><br/>reads the request, decides which tools to call"]
subgraph WG["agentic-sim-world-gen (this repository): MCP server + skill, the same operations from the CLI"]
direction LR
S["Search and pick assets"] --> L["Lay out<br/>relations, arrays, shelves"] --> V["Review in the browser<br/>thumbnails, top view"] --> U["Write the USD world<br/>and LICENSES.md"]
end
SRC["<b>Asset sources</b><br/>SimReady Central<br/>Isaac Sim bundled assets<br/>Poly Haven CC0<br/>your USD folders"]
ISAAC["<b>Isaac Sim 6.1</b><br/>runs the check script:<br/>physics, screenshots"]
IN --> CC
CC -- "calls the tools" --> WG
SRC -- "searched" --> WG
WG -- "USD" --> ISAAC
ISAAC -- "report: fix and check again" --> CC
style WG fill:#536dfe14,stroke:#536dfe,stroke-width:3px
classDef repo fill:#536dfe2e,stroke:#536dfe
classDef input fill:#9e9e9e26,stroke:#9e9e9e,stroke-dasharray:5 5
classDef claude fill:#d977572e,stroke:#d97757,stroke-dasharray:5 5
classDef assets fill:#e3a0082e,stroke:#e3a008,stroke-dasharray:5 5
classDef isaac fill:#76b9002e,stroke:#76b900,stroke-dasharray:5 5
class S,L,V,U repo
class IN input
class CC claude
class SRC assets
class ISAAC isaacBlue with a thick frame: this repository. Dashed: outside it (orange: Claude Code, yellow: asset sources, green: Isaac Sim, gray: your input).
This is a personal project. It is not an official NVIDIA product and is not sponsored or endorsed by NVIDIA. Isaac Sim, Omniverse, SimReady and Replicator are trademarks of NVIDIA Corporation.
SimReady Central is searched through the unpublished API its web page uses. It is unofficial and may change
or stop working without notice. The official endpoint (search.simready.omniverse.nvidia.com, used by the
simready-search package) indexes a different catalog, mostly the Isaac Sim bundled assets, and does not
return the SimReady Central datasets on Hugging Face, so it cannot replace this API.
Asset licenses differ per asset, and non-commercial ones (e.g. CC-BY-NC) can be picked. Check theLICENSES.md written for each scene before you use or share it. The Isaac Sim bundled assets are licensed
by NVIDIA and are not covered by this repository's license; handing out a USD with assets packed in is
redistribution.
Do not use photos or videos that show people as input.
Demo: a warehouse

One turn, asked in Japanese, waits fast-forwarded: "Make a 24 m × 16 m warehouse. Put pallet racks in 4 rows with
aisles between them and cardboard boxes on the shelves; pallets and a forklift in the loading area at the front.
Check it with physics in Isaac Sim and sum up the licenses." Claude fills in the defaults, searches and lays out
everything at once, swaps the stocked racks for empty ones from the candidates (select_asset) and puts boxes on
the shelves (set_fill). The physics check in Isaac Sim 6.1 finds 2 boxes that slid and a forklift that moved:
Claude spaces the boxes out, tries three forklifts (their seat or handle parts come loose) and settles on a walk-behind
electric stacker, and the check passes (221 objects). It ends with the request checklist, the licenses (all
CC BY 4.0) and a view from 24 directions. Assets from SimReady Central only
(AGENTIC_SIM_WORLD_GEN_SOURCES=simready-central, AGENTIC_SIM_WORLD_GEN_MDL=0). A screen recording on Ubuntu
(Isaac Sim not opened, the check runs headless); full quality: docs/videos/warehouse_demo.mp4.
Assets (all CC BY 4.0, details in LICENSES.md): "sm_rack_h3m_s02_01", "sm_box_cardboard_b02_01" and "sm_vehicle_forklift_orange_a01_01" by NVIDIA (nvidia/simready-assets), "pallet" by ForgeCAD (ForgeCAD/simready-assets), CC BY 4.0.
One run end to end (the example in Getting Started; ②–④ are real screens):

Related MCP server: IsaacSim-MCP
Features
MCP server + skill for Claude Code (63 tools); the same operations from the CLI
agentic-sim-world-genInput: text (rooms, warehouses, outdoor sites), floor plans (JSON / DXF / IFC, or traced from an image), photos and walk-through videos
Layout: relations (against a wall, in front of, next to, on, facing), arrays (rack rows with aisles), stocking shelves, overlap / out-of-room / door clearance checks
Look: floor and wall materials (Isaac Sim MDL or CC0 PBR), lighting presets, moods, outdoor ground / roads / buildings / sky
Review: candidate thumbnails and a top-down layout in a browser preview; physics check and screenshots in Isaac Sim 6.1 (headless or live)
Synthetic data with Replicator: RGB, 2D boxes, segmentation, depth (COCO / KITTI)
Licenses: source and license of every asset recorded in the USD and in
LICENSES.md
Objects are placed by relation, not by coordinates; the solver works out the position and the direction, and keeps them when an asset is swapped for one of another size:
When no asset fits, a close one is used instead and reported (no 3D generation). Your own asset folders can be
added with local:.
Requirements
Tested | Not tested | |
Isaac Sim | 6.1 pip install, Ubuntu 24.04, RTX 2080 Ti | the standalone (binary) build, other versions, Windows |
The rest (search, preview, USD) | Ubuntu 24.04 (Python 3.14), macOS on Apple Silicon (Python 3.12) | Windows |
Python 3.12+ and uv; Claude Code to use it as an agent; Isaac Sim 6.1 only for the physics check, screenshots and synthetic data. Without a GPU you can still search, preview and write USD.
Install
git clone https://github.com/atinfinity/agentic-sim-world-gen.git
cd agentic-sim-world-gen
uv sync --extra usd --extra hf --extra mcp --extra dev
source .venv/bin/activate
agentic-sim-world-gen resolve examples/p0_study_room.json -o out/study # search -> candidates.html, layout.svg
agentic-sim-world-gen compose out/study/scene.resolved.json # -> out/study/scene.usda
agentic-sim-world-gen check out/study # physics check in Isaac Sim (Linux)Install usd-core only in this venv, never in Isaac Sim's Python. Point to Isaac Sim with ISAACSIM_PYTHON
(pip install) or ISAACSIM_PATH (default ~/isaacsim), and set OMNI_KIT_ACCEPT_EULA=YES to run it non-interactively.
The other environment variables are in Getting Started "3. Environment variables".
With Claude Code: .mcp.json registers the agentic-sim-world-gen MCP server and .claude/skills/world-gen
is the skill. To only make worlds, create a workspace outside the repository with
agentic-sim-world-gen workspace <dir>, start claude there and ask, e.g. "Make a 10 m² study with a desk,
a chair and a bookshelf".
Guides
To use it:
Guide | |
from the install to your first world | |
Ubuntu with an RTX GPU and a screen (live mode too) | |
the GPU machine is an Ubuntu box on the LAN; you use ssh and a browser from a Mac or similar |
To change the code:
Guide | |
the flow, data, search, layout, checks, USD, Isaac Sim, the conversation and rooms from photos, with diagrams | |
setup, tests, where things are, pull requests, the project page, GitHub Actions |
The Japanese versions are in docs/.
License
The code is under the Apache License 2.0 (see NOTICE). Not covered by it:
Assets are not included. A generated world refers to assets that keep their own licenses (the scene's
LICENSES.md).Images (
docs/images,src/agentic_sim_world_gen/data/moods) show assets under their own licenses (the demo's attribution is above).Bundled indexes (
src/agentic_sim_world_gen/data/*.gz) are lists of names, sizes and URLs; their origin is in NOTICE.Example photos (
examples/photos) are CC0 from Wikimedia Commons (SOURCES.md).Optional dependency: IFC import (extra
ifc) uses IfcOpenShell, LGPL-3.0-or-later (imported, not bundled).Optional models (not bundled; downloaded only when you allow it): SigLIP 2 for the taste check (Apache-2.0), MoGe-2 for photo depth (MIT).
Contributions: Contributing. Security: SECURITY.md.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
This server cannot be deployed
Maintenance
Related MCP Connectors
Generate 3D models, worlds, PBR materials, and coordinated asset packs from AI agents.
A shared 3D world that agents build in. Look around, claim a space, place things, propose edits.
A shared, persistent workspace where people and agents build real things together.
Build, validate, and deploy multi-agent AI solutions from any AI environment.
Related MCP Servers
- AlicenseNot gradedqualityCmaintenanceEnables AI agents to build and edit Roblox Studio places by inspecting, generating terrain, geometry, and code, with validation and iteration.MIT
- AlicenseCqualityCmaintenanceEnables AI assistants to control NVIDIA Isaac Sim by building scenes, loading assets, operating robots and humans, reading sensors, managing simulation, and creating Action Graphs.1291MIT
- AlicenseNot gradedqualityBmaintenanceEnables a shared 3D workspace where humans and AI agents co-design in real time, with scene awareness, goal-level tools, visible proposals, deterministic reviews, and approval-gated changes.MIT
- AlicenseCqualityBmaintenanceEnables agents to discover and acquire USD assets on demand, compose and edit scenes, run Isaac Sim simulations, record camera output, and validate results through evidence-driven checks.5239Apache 2.0