agentic3d

An agent for instruction-guided editing of 3D indoor scenes. Given a scene mesh and a natural-language instruction such as "remove the table and move the chair to where it was" or "add a sofa against the wall", the system localizes the objects the instruction refers to, applies the corresponding geometric edits, and checks the result against the instruction from rendered views.

The implementation runs on CPU or Apple Silicon (MPS) with no CUDA dependency: rendering is done with ray casting and a lightweight shading pass rather than a GPU rasterizer, object localization uses frozen open-vocabulary 2D models (Grounding DINO and SAM), and meshes for "add" operations are retrieved from a local Objaverse index.

Method

The pipeline is organized into the following stages, each implemented as a module in agentic3d/:

1. Rendering (render). Multi-view ray casting produces, per view, a depth map, a per-pixel hit-face index, and a shaded RGB image. The hit-face index makes the mapping from image pixels back to mesh faces exact.

2. 2D segmentation (segment2d). Grounding DINO proposes boxes for the target phrases and SAM converts them to masks, on each rendered view.

3. Mask lifting (lift). Per-view masks are back-projected onto mesh faces and aggregated by voting across views; the vote field is thresholded, cleaned, and split into connected components to yield one submesh per object instance.

4. Scene editing (edits). Instances populate an editable scene graph that supports add, remove, move, rotate, scale, and replace operations, with floor snapping and axis-aligned collision handling. Composing the graph produces the output mesh.

5. Agent loop (agent). A language model plans a sequence of edit operations from the instruction and the scene description, the operations are executed, and a vision-language model verifies the rendered result. Failed verifications trigger revision and re-planning.

Objects for "add" operations are supplied by retrieval, which matches the query to Objaverse-LVIS categories by CLIP text similarity and re-ranks a small set of candidate meshes by CLIP image similarity.

Related MCP server: Hayba

Installation

conda env create -f environment.yml
conda activate agentic3d
pip install -e .
python -m ipykernel install --user --name agentic3d --display-name "Python (agentic3d)"

The agent and evaluation code read an OpenAI API key from a .env file at the repository root (OPENAI_API_KEY=...).

Usage

A single instruction is applied with pipeline.edit_scene:

from agentic3d import pipeline

edited_mesh, scene_graph, trajectory = pipeline.edit_scene(mesh, "add a sofa against the wall")
edited_mesh.export("edited.glb")

mesh is a file path or a trimesh.Trimesh. The returned trajectory records the planned operations, each operator's report, and the verifier's verdict.

EditSession applies several instructions to the same scene, reusing the segmentation and scene graph between calls:

from agentic3d.pipeline import EditSession

session = EditSession(mesh, targets=["chair", "table"])
session.apply("remove the table")
session.apply("add a sofa where the table was")
session.mesh().export("edited.glb")

MCP server

agentic3d/mcp_server.py exposes the scene-editing pipeline over the Model Context Protocol as a stdio server, so an MCP client (for example Claude Desktop or the MCP Inspector) can drive it directly. The tools are load_scene, segment_scene, get_scene_info, add_object, remove_object, move_object, rotate_object, scale_object, replace_object, undo, render_scene (returns a rendered image), save_scene, and edit_with_agent, which runs the full plan / apply / verify / revise loop for one instruction with an internal model and returns a summary and a render. Two prompt templates are also provided: edit_scene(scene_path, instruction) and describe_scene(scene_path).

After pip install -e . the server is on the path as agentic3d-mcp (equivalently python -m agentic3d.mcp_server). Point an MCP client at the environment's executable directly:

{
  "mcpServers": {
    "agentic3d": {
      "command": "/path/to/conda/envs/agentic3d/bin/agentic3d-mcp"
    }
  }
}

conda run also works but needs --no-capture-output, otherwise it buffers the server's stdio and the connection never completes:

{
  "mcpServers": {
    "agentic3d": {
      "command": "conda",
      "args": ["run", "--no-capture-output", "-n", "agentic3d", "agentic3d-mcp"]
    }
  }
}

Or inspect it interactively:

npx @modelcontextprotocol/inspector conda run --no-capture-output -n agentic3d agentic3d-mcp

scripts/make_demo_scene.py writes a room.glb (a floor, two walls, and a chair, table, and potted plant retrieved from Objaverse) to try the server against:

python scripts/make_demo_scene.py
agentic3d-mcp --test room.glb chair table plant

A typical session calls load_scene, then segment_scene(["chair", "table", "plant"]), then the edit tools, then render_scene and save_scene. The client's own model reads the images returned by render_scene to decide whether the edit matched the request.

Evaluation

eval.evaluate_edit scores an edit:

from agentic3d import eval

scores = eval.evaluate_edit(input_mesh, edited_mesh, instruction)

By default the score is reference-free: a vision-language model rates the result from 0 to 5 on instruction adherence, physical plausibility, and spatial layout. Passing target_mesh= adds Chamfer distance and axis-aligned bounding-box IoU against a hand-authored target. The evaluation notebook runs a small sweep of tasks and plots the resulting scores.

Repository structure

agentic3d/               library modules (see Method)
agentic3d/mcp_server.py  Model Context Protocol server
notebooks/               one notebook per module, for inspecting intermediate results
scripts/                 helper scripts (demo scene generation)
pyproject.toml           package metadata and the agentic3d-mcp entry point

The notebooks depend on the agentic3d package and on the embreex ray-mesh intersection backend that the project environment installs.