# SpatialLens

**USD/USDZ/USDA scene graph parser and spatial analysis tool for AI agents.**

SpatialLens gives AI coding agents (Claude, Codex, Gemini, etc.) structured access to 3D scene data from Apple's USD ecosystem. It parses binary `.usdz` and text `.usda` files into structured JSON that any AI can reason about — entity hierarchies, material shader graphs, spatial positions, transforms, and relationships between objects.

> **Status: Work in Progress.** Actively developed and tested against real visionOS game assets. Core parsing and spatial analysis are functional. Contributions and feedback welcome.

---

## Why This Exists

AI agents working on visionOS, RealityKit, or any USD-based 3D project are blind to what's actually in the asset files. They can read Swift code that references entity names and positions, but they can't see the USDZ contents — the hierarchy, the materials, the transforms, the shader graphs. This creates a gap where the AI guesses instead of knowing.

SpatialLens closes that gap by translating binary 3D assets into structured text that AI agents natively understand.

---

## Two Ways to Use It

### 1. CLI (any AI agent, any environment)

```bash
python cli.py <command> <file> [options]
```

No MCP required. Runs anywhere with Python 3.12+ and Apple's USD tools. Output is JSON to stdout — pipe it, save it, paste it into a prompt.

### 2. MCP Server (Claude Desktop, or any MCP-compatible client)

```bash
python server.py
```

Exposes all tools over MCP for real-time use during coding sessions.

---

## Prerequisites

- **macOS** (required for Apple USD tools)
- **Python 3.12+**
- **Apple USD CLI tools** — `usdcat` and `usdtree` at `/usr/bin/` (ships with Xcode Command Line Tools)

Verify your setup:
```bash
usdcat --version    # Should print "Apple USD Tools (x.x.x)"
usdtree --version   # Same
python3 --version   # 3.12+
```

---

## Setup

```bash
git clone https://github.com/RevivrStudios/spatiallens-mcp.git
cd spatiallens-mcp
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
```

---

## CLI Commands

All commands output structured JSON to stdout unless noted.

### `scene` — Full Scene Graph
```bash
python cli.py scene Player_Ships.usdz
```
Returns the complete prim hierarchy with types, paths, properties, and children. Use this for deep analysis.

### `tree` — Quick Hierarchy (text output)
```bash
python cli.py tree Player_Ships.usdz
```
Returns an indented text tree. Fast overview of what's in a file.

### `materials` — Materials and Shader Graphs
```bash
python cli.py materials TerrainGlass.usda
```
Returns all materials with shader node IDs, input parameters, output connections, and the full shader graph. Use this for debugging visual issues.

### `transforms` — Positioned Entities
```bash
python cli.py transforms Player_Ships.usdz
```
Returns every entity that has translate, rotate, or scale data.

### `entities` — List All Entities
```bash
python cli.py entities Enemy_Ships.usdz
python cli.py entities Enemy_Ships.usdz --type Mesh
```
Flat list of all entities. Filter by type: `Mesh`, `Material`, `Shader`, `Xform`, `Sphere`, `Scope`.

### `details` — Single Entity Deep Dive
```bash
python cli.py details Player_Ships.usdz --path "/__spaceships_set/Meshes"
```
Full properties and children for one specific entity. If the path doesn't exist, returns all available paths.

### `layout` — Spatial Layout Analysis
```bash
python cli.py layout Player_Ships.usdz
```
Returns:
- All positioned entities with coordinates
- Bounding box (min, max, center, size in meters)
- 5 nearest entity pairs with distances
- Plain English spread description (compact / room-scale / large-scale)
- Coordinate convention reference

### `diff` — Compare Two Files
```bash
python cli.py diff Player_Ships.usdz Enemy_Ships.usdz
```
Structural diff: added prims, removed prims, changed properties. Use after editing a USDZ in Reality Composer Pro.

### `overlaps` — Find Overlapping Entities
```bash
python cli.py overlaps Player_Ships.usdz --threshold 0.05
```
Flags entity pairs closer than the threshold (default 1cm). Catches z-fighting, stacked geometry, accidental overlaps.

### `relationship` — Spatial Relationship Between Two Entities
```bash
python cli.py relationship Player_Ships.usdz \
    --a "/__spaceships_set/.../SpaceShip1_" \
    --b "/__spaceships_set/.../SpaceShip2_"
```
Returns distance and plain English description: "SpaceShip2_ is 1.665m to the right and 0.022m toward viewer from SpaceShip1_"

### `move` — Compute New Position
```bash
python cli.py move Player_Ships.usdz \
    --path "/__spaceships_set/.../SpaceShip1_" \
    --direction right \
    --meters 0.5
```
Returns current position, direction, and computed new position. Directions: `left`, `right`, `up`, `down`, `forward`, `back`, `toward viewer`, `away from viewer`.

---

## Coordinate Convention

SpatialLens uses the RealityKit / visionOS coordinate system:

| Direction | Axis | Sign |
|-----------|------|------|
| Right | X | + |
| Left | X | - |
| Up | Y | + |
| Down | Y | - |
| Toward viewer | Z | + |
| Away from viewer | Z | - |

**Units:** 1 unit = `metersPerUnit` meters (read from the USD file metadata). Common values:
- `metersPerUnit = 1` — units are meters (RealityKit default)
- `metersPerUnit = 0.01` — units are centimeters (common in Sketchfab/Blender exports)

SpatialLens reads `metersPerUnit` from each file and converts all spatial output to meters automatically.

---

## MCP Server Setup

For Claude Desktop, add to your `claude_desktop_config.json`:

```json
{
  "mcpServers": {
    "spatiallens": {
      "command": "/path/to/spatiallens-mcp/venv/bin/python",
      "args": ["/path/to/spatiallens-mcp/server.py"]
    }
  }
}
```

The MCP server exposes 11 tools:
- `parse_scene_graph` — full scene graph
- `hierarchy_tree` — quick text tree
- `get_materials` — materials and shaders
- `get_transforms` — positioned entities
- `list_entities` — flat entity list with type filter
- `get_entity_details` — single entity deep dive
- `diff_scenes` — structural diff
- `spatial_layout` — spatial analysis with bounding box
- `spatial_relationship` — relationship between two entities
- `check_spatial_overlaps` — overlap detection
- `move_suggestion` — compute new position from direction

---

## Project Structure

```
spatiallens-mcp/
├── cli.py              # Standalone CLI (no MCP needed)
├── server.py           # MCP server entry point
├── usda_parser.py      # USDA text → structured dict parser
├── spatial_math.py     # Coordinate geometry and spatial analysis
├── requirements.txt    # Python dependencies (fastmcp, mcp)
└── tests/
    ├── test_parser.py  # Unit tests for USDA parser
    └── test_tools.py   # Integration tests against real USDZ files
```

**Core modules:**

- **`usda_parser.py`** — Line-by-line state machine that parses the text output of `usdcat --flatten`. Extracts prim hierarchy, types, properties, material bindings, shader connections. Handles nested metadata blocks and customData annotations.

- **`spatial_math.py`** — Pure math operations on parsed prim data. Position extraction (from `xformOp:translate` or 4x4 transform matrices), distance calculation, relative position descriptions, bounding box analysis, overlap detection, move suggestions. No app-specific logic.

- **`server.py`** — FastMCP wrapper that shells out to `usdcat`/`usdtree`, passes results through the parser and spatial math modules, and returns structured JSON.

- **`cli.py`** — Argparse-based CLI that uses the same parser and spatial math modules. Identical output, no MCP dependency.

---

## For AI Agents: How to Use This

If you're an AI agent reading this README, here's how to get value from SpatialLens:

1. **Clone and set up** the repo (see Setup above)
2. **Run `python cli.py tree <file>`** first to see what's in a USDZ file
3. **Run `python cli.py layout <file>`** to understand spatial positioning
4. **Run `python cli.py materials <file>`** when debugging visual issues
5. **Run `python cli.py entities <file> --type Mesh`** to find specific geometry
6. **Run `python cli.py diff <old> <new>`** after editing a scene to verify changes

The JSON output is designed to be parsed programmatically. Every command returns valid JSON to stdout (except `tree` which returns text).

**Key insight:** When the human says "move it to the right" or "the ships are too close together," you can now quantify that with real coordinates instead of guessing. Use `layout` to see where things are, `relationship` to measure distances, and `move` to compute exact position changes.

---

## Tested Against

- **Player_Ships.usdz** — 5 spaceship models, 25 entities, centimeter-scale (metersPerUnit=0.01)
- **Enemy_Ships.usdz** — 12 enemy ship models, 46 entities, centimeter-scale
- **TerrainGlass.usda** — Glass material with 14-node shader graph (RealityKit MaterialX)

All from a real visionOS game project. 17/17 tests passing.

---

## Limitations

- **macOS only** — requires Apple's `usdcat` and `usdtree` CLI tools
- **Local transforms only** — positions are local to the prim, not accumulated world-space (parent transforms not yet composed)
- **No mesh geometry** — parses hierarchy, properties, and materials, but does not read vertex/face data
- **No texture content** — identifies texture file references but cannot read image data
- **Parser handles common USDA patterns** — exotic USD features (variants, payloads, instancing) may not be fully parsed

---

## License

MIT