# agentic3d

An agent for instruction-guided editing of 3D indoor scenes. Given a scene mesh and a natural-language instruction such as *"remove the table and move the chair to where it was"* or *"add a sofa against the wall"*, the system localizes the objects the instruction refers to, applies the corresponding geometric edits, and checks the result against the instruction from rendered views.

The implementation runs on CPU or Apple Silicon (MPS) with no CUDA dependency: rendering is done with ray casting and a lightweight shading pass rather than a GPU rasterizer, object localization uses frozen open-vocabulary 2D models (Grounding DINO and SAM), and meshes for "add" operations are retrieved from a local Objaverse index.

## Method

The pipeline is organized into the following stages, each implemented as a module in `agentic3d/`:

1. **Rendering** (`render`). Multi-view ray casting produces, per view, a depth map, a per-pixel hit-face index, and a shaded RGB image. The hit-face index makes the mapping from image pixels back to mesh faces exact.
2. **2D segmentation** (`segment2d`). Grounding DINO proposes boxes for the target phrases and SAM converts them to masks, on each rendered view.
3. **Mask lifting** (`lift`). Per-view masks are back-projected onto mesh faces and aggregated by voting across views; the vote field is thresholded, cleaned, and split into connected components to yield one submesh per object instance.
4. **Scene editing** (`edits`). Instances populate an editable scene graph that supports add, remove, move, rotate, scale, and replace operations, with floor snapping and axis-aligned collision handling. Composing the graph produces the output mesh.
5. **Agent loop** (`agent`). A language model plans a sequence of edit operations from the instruction and the scene description, the operations are executed, and a vision-language model verifies the rendered result. Failed verifications trigger revision and re-planning.

Objects for "add" operations are supplied by `retrieval`, which matches the query to Objaverse-LVIS categories by CLIP text similarity and re-ranks a small set of candidate meshes by CLIP image similarity.

## Installation

```bash
conda env create -f environment.yml
conda activate agentic3d
pip install -e .
python -m ipykernel install --user --name agentic3d --display-name "Python (agentic3d)"
```

The agent and evaluation code read an OpenAI API key from a `.env` file at the repository root (`OPENAI_API_KEY=...`).

## Usage

A single instruction is applied with `pipeline.edit_scene`:

```python
from agentic3d import pipeline

edited_mesh, scene_graph, trajectory = pipeline.edit_scene(mesh, "add a sofa against the wall")
edited_mesh.export("edited.glb")
```

`mesh` is a file path or a `trimesh.Trimesh`. The returned `trajectory` records the planned operations, each operator's report, and the verifier's verdict.

`EditSession` applies several instructions to the same scene, reusing the segmentation and scene graph between calls:

```python
from agentic3d.pipeline import EditSession

session = EditSession(mesh, targets=["chair", "table"])
session.apply("remove the table")
session.apply("add a sofa where the table was")
session.mesh().export("edited.glb")
```

## MCP server

`agentic3d/mcp_server.py` exposes the scene-editing pipeline over the Model Context Protocol as a stdio server, so an MCP client (for example Claude Desktop or the MCP Inspector) can drive it directly. The tools are `load_scene`, `segment_scene`, `get_scene_info`, `add_object`, `remove_object`, `move_object`, `rotate_object`, `scale_object`, `replace_object`, `undo`, `render_scene` (returns a rendered image), `save_scene`, and `edit_with_agent`, which runs the full plan / apply / verify / revise loop for one instruction with an internal model and returns a summary and a render. Two prompt templates are also provided: `edit_scene(scene_path, instruction)` and `describe_scene(scene_path)`.

After `pip install -e .` the server is on the path as `agentic3d-mcp` (equivalently `python -m agentic3d.mcp_server`). Point an MCP client at the environment's executable directly:

```json
{
  "mcpServers": {
    "agentic3d": {
      "command": "/path/to/conda/envs/agentic3d/bin/agentic3d-mcp"
    }
  }
}
```

`conda run` also works but needs `--no-capture-output`, otherwise it buffers the server's stdio and the connection never completes:

```json
{
  "mcpServers": {
    "agentic3d": {
      "command": "conda",
      "args": ["run", "--no-capture-output", "-n", "agentic3d", "agentic3d-mcp"]
    }
  }
}
```

Or inspect it interactively:

```bash
npx @modelcontextprotocol/inspector conda run --no-capture-output -n agentic3d agentic3d-mcp
```

`scripts/make_demo_scene.py` writes a `room.glb` (a floor, two walls, and a chair, table, and potted plant retrieved from Objaverse) to try the server against:

```bash
python scripts/make_demo_scene.py
agentic3d-mcp --test room.glb chair table plant
```

A typical session calls `load_scene`, then `segment_scene(["chair", "table", "plant"])`, then the edit tools, then `render_scene` and `save_scene`. The client's own model reads the images returned by `render_scene` to decide whether the edit matched the request.

## Evaluation

`eval.evaluate_edit` scores an edit:

```python
from agentic3d import eval

scores = eval.evaluate_edit(input_mesh, edited_mesh, instruction)
```

By default the score is reference-free: a vision-language model rates the result from 0 to 5 on instruction adherence, physical plausibility, and spatial layout. Passing `target_mesh=` adds Chamfer distance and axis-aligned bounding-box IoU against a hand-authored target. The `evaluation` notebook runs a small sweep of tasks and plots the resulting scores.

## Repository structure

```
agentic3d/               library modules (see Method)
agentic3d/mcp_server.py  Model Context Protocol server
notebooks/               one notebook per module, for inspecting intermediate results
scripts/                 helper scripts (demo scene generation)
pyproject.toml           package metadata and the agentic3d-mcp entry point
```

The notebooks depend on the `agentic3d` package and on the `embreex` ray-mesh intersection backend that the project environment installs.