# babylonjs-inspector-mcp

[![npm version](https://img.shields.io/npm/v/babylonjs-inspector-mcp.svg)](https://www.npmjs.com/package/babylonjs-inspector-mcp)
[![CI Status](https://github.com/apowers313/babylonjs-inspector-mcp/actions/workflows/ci.yml/badge.svg)](https://github.com/apowers313/babylonjs-inspector-mcp/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

An MCP (Model Context Protocol) server that lets an LLM inspect a running BabylonJS 3D scene.

> **Status: early development (0.x).** The MCP server, the tool surface, the WebSocket
> protocol and the HTTP server are implemented. The browser-side client that actually talks
> to your BabylonJS scene is still a stub, so today the tools connect and validate but
> return "no scene connected" until that lands. Read
> [Implementation status](#implementation-status) for the exact breakdown before you
> install this.

## The problem

When an LLM writes BabylonJS code -- shaders, geometry, materials, lighting, animations,
post-processing -- it has no way to check the result. It cannot:

- See what the scene looks like, from any angle
- Tell whether two objects are actually where it intended them to be
- Read back shader compilation results or WebGL state
- Confirm that a material or light produces the effect it was asked for
- Debug a rendering artifact it cannot observe

The LLM writes code, the human looks at the screen, and the human becomes the LLM's eyes.

## The solution

An MCP server that gives the LLM three kinds of access to a live scene:

1. **Scene data** -- structured JSON from BabylonJS: scene graph, meshes, materials, cameras,
   lights, animations, textures, physics, post-processing, performance counters.
2. **WebGL introspection** -- Spector.js frame captures: draw calls, shader source, WebGL
   state, texture bindings.
3. **VLM scene understanding** -- Gemini vision analysis of screenshots, depth maps and
   multi-view renders, combined with scene metadata (bring your own API key).

Plus a `scene_evaluate` escape hatch that runs arbitrary JavaScript against the scene, in the
spirit of Playwright's `browser_evaluate`.

## Architecture

The server is a Node.js process started by your MCP host. It speaks MCP over stdio to the host
and WebSocket to your browser. It does not embed a browser; your app connects to it.

```
  +---------------------------+
  | MCP client                |   Claude Code, Claude Desktop, or any MCP host
  +---------------------------+
              ^
              |  stdio (JSON-RPC) -- MCP tool calls and results
              v
  +---------------------------+
  | babylonjs-inspector-mcp   |   Node.js process spawned by the MCP host
  |                           |   - registers the MCP tools
  |                           |   - BrowserBridge: correlates requests by UUID
  |                           |   - HTTP + WebSocket server on port 9070
  +---------------------------+
              ^
              |  WebSocket (ws://localhost:9070) -- commands and responses
              v
  +---------------------------+
  | Browser                   |   your Vite dev server, or any page
  |   inspector client        |   - discovers the BabylonJS engine
  |   + your BabylonJS scene  |   - runs commands against the live scene
  +---------------------------+
```

A tool call travels the whole chain: the host calls `babylonjs_scene_inspect`, the server
dispatches it to the `BrowserBridge`, the bridge sends a WebSocket command tagged with a UUID,
the browser client executes it against the scene, and the response comes back up the chain.

Only one browser connection is tracked at a time. Every operation has a timeout tier:

| Operation | Timeout |
|-----------|---------|
| Scene query / inspect | 5s |
| Screenshot capture | 15s |
| Spector.js frame capture | 30s |
| Scene evaluate | 30s |
| Multi-view capture | 45s |
| Scene export | 60s |

## Requirements

- Node.js >= 20
- A BabylonJS application you can run locally
- Optionally, a Gemini API key for the VLM analysis tool

## Installation

```bash
npx -y babylonjs-inspector-mcp                  # run on demand
npm install -g babylonjs-inspector-mcp          # binary on your PATH
npm install --save-dev babylonjs-inspector-mcp  # Vite plugin / browser client
```

To work from a checkout instead:

```bash
git clone https://github.com/apowers313/babylonjs-inspector-mcp.git
cd babylonjs-inspector-mcp
npm install
npm run build
```

That produces `dist/`, which `bin/babylonjs-inspector-mcp.js` needs in order to run.

## MCP client configuration

Add the server to your MCP settings, pointing at your local build. For Claude Code:

```bash
claude mcp add babylonjs-inspector -- node /path/to/babylonjs-inspector-mcp/bin/babylonjs-inspector-mcp.js
```

Or configure it by hand -- this is the JSON block for
`claude_desktop_config.json`, `.mcp.json`, or your host's equivalent:

```json
{
  "mcpServers": {
    "babylonjs-inspector": {
      "command": "node",
      "args": ["/path/to/babylonjs-inspector-mcp/bin/babylonjs-inspector-mcp.js"],
      "env": {
        "BABYLONJS_INSPECTOR_PORT": "9070",
        "GEMINI_API_KEY": "your-key-here"
      }
    }
  }
}
```

After the package is published, `"command": "npx"` with
`"args": ["-y", "babylonjs-inspector-mcp"]` replaces those two lines, and a global
install lets you use `"command": "babylonjs-inspector-mcp"` with no `args`. Use `npx -y` so
that npx's install prompt can never end up on the stdio channel the MCP protocol is using.

`GEMINI_API_KEY` is optional. When it is absent, the `babylonjs_scene_analyze` tool is not
registered at all, so it never shows up in the LLM's tool list. When it is present, the tool
appears. Everything else works without it.

## Browser-side setup

The server needs your BabylonJS page to connect back to it over WebSocket. There are two
supported ways to make that happen.

### Vite plugin (zero instrumentation)

Add the plugin to your app's `vite.config.ts`. It only applies in dev mode (`apply: "serve"`),
so production builds are untouched, and it needs no changes to your application code.

```typescript
import { defineConfig } from "vite";
import { babylonjsInspector } from "babylonjs-inspector-mcp/vite";

export default defineConfig({
  plugins: [
    babylonjsInspector({
      port: 9070,        // must match BABYLONJS_INSPECTOR_PORT; default 9070
      autoConnect: true, // connect on page load; default true
    }),
  ],
});
```

The plugin appends a `<script type="module">` tag to your HTML that imports the inspector
client from `http://localhost:<port>/inspector-client.js`, which the MCP server serves, and
assigns the instance to `window.__babylonjsInspector`.

### Manual import

If you are not on Vite, or you want explicit control over when the inspector attaches:

```typescript
import { InspectorClient } from "babylonjs-inspector-mcp/client";

const inspector = new InspectorClient({
  port: 9070,         // default 9070
  autoDiscover: true, // find the BabylonJS engine automatically; default true
  scene: myScene,     // optional: pass the scene explicitly instead
});
```

Both paths are wired up but not yet functional -- see below.

## MCP tools

Six tools are registered. Five are always present; `babylonjs_scene_analyze` is registered
only when `GEMINI_API_KEY` is set.

| Tool | Returns | Needs a live browser connection |
|------|---------|--------------------------------|
| `babylonjs_connection_status` | Connection state, engine type, canvas size, and which tool categories are currently usable | No |
| `babylonjs_scene_inspect` | Scene data for one of 15 categories, from a counts-only summary to a full scene graph | Yes |
| `babylonjs_scene_evaluate` | The JSON-serialized return value of a JavaScript function run against the scene | Yes |
| `babylonjs_screenshot` | A map of view name to saved image file path | Yes, and the engine must be able to render |
| `babylonjs_spector_capture` | Spector.js WebGL frame data: draw calls, shader source, GL state | Yes, plus WebGL and Spector.js in the page |
| `babylonjs_scene_analyze` | A Gemini vision answer to a question about the rendered scene | Yes, rendering capable, plus `GEMINI_API_KEY` |

Every browser-dependent tool degrades to a plain JSON `{ "error": "..." }` message rather than
throwing, so the LLM gets a readable reason instead of a protocol failure.

### babylonjs_connection_status

No parameters. Returns `connected`, `engineType`, `canRender`, `hasSpector`, `canvasSize`, and
an `availableTools` object reporting which of `sceneInspect`, `sceneEvaluate`, `screenshot`,
`spectorCapture` and `sceneAnalyze` are usable right now. Every entry is `false` while no
browser is attached, because every one of those tools answers "No BabylonJS scene connected"
in that state. Call this first when a tool reports that nothing is connected.

### babylonjs_scene_inspect

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `category` | enum | `summary` | What to inspect |
| `target` | string | -- | Name or ID of a specific mesh/material/light/texture, for the `*_detail` categories |
| `filter` | string | -- | Glob pattern matched against names, e.g. `wall*` or `*metal*` |

`category` accepts: `summary`, `full`, `meshes`, `mesh_detail`, `materials`,
`material_detail`, `cameras`, `lights`, `animations`, `textures`, `physics`,
`post_processing`, `performance`, `scene_tree`, `webgl_errors`.

`summary` and `full` return an overview of the whole scene; every other category drills into
one aspect. This is deliberately one parameterized tool rather than fifteen separate ones, to
keep the host's tool list short and the LLM's choice simple.

### babylonjs_scene_evaluate

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `function` | string | required | A JavaScript function body receiving `(scene, engine, BABYLON)` |

Example argument:

```javascript
(scene) => scene.meshes.map((m) => ({ name: m.name, pos: m.position.asArray() }))
```

The return value is serialized to JSON. This is a development tool with no sandbox -- the code
runs with full access to the page.

### babylonjs_screenshot

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `views` | array of enum | `["current"]` | Which views to capture |
| `customCamera` | object | -- | `{ position: [x,y,z], target: [x,y,z], fov?, projection? }` |
| `width` | number | `1024` | Image width in pixels |
| `height` | number | `1024` | Image height in pixels |
| `format` | `png` or `jpeg` | `png` | Image format |

`views` accepts: `current`, `front`, `top`, `side`, `isometric`, `depth`, `wireframe`,
`annotated`. `customCamera.projection` is `perspective` or `orthographic`.

Images are written to disk and the tool returns file paths, not base64, so a multi-view capture
does not flood the context window.

### babylonjs_spector_capture

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `detailLevel` | `summary`, `shaders`, `full` | `summary` | How much of the capture to return |
| `quickCapture` | boolean | `true` | Skip per-draw-call thumbnails for a faster capture |

`summary` returns counts and timings, `shaders` adds deduplicated shader source, `full` adds
per-draw-call WebGL state.

### babylonjs_scene_analyze

Registered only when `GEMINI_API_KEY` is set.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `question` | string | required | A specific question about the rendered scene |
| `views` | array of enum | `["current", "depth", "top"]` | Which renders to send to the model |
| `includeSceneMetadata` | boolean | `true` | Send structured scene data alongside the images |
| `resolution` | `low`, `medium`, `high` | `medium` | 512, 1024 or 2048 pixels |

`views` accepts: `current`, `front`, `top`, `side`, `isometric`, `depth`, `annotated`.

Good questions are concrete: "Are the shadows rendering correctly?", "Is the sphere positioned
above the plane?", "Does the PBR material look metallic?"

## Configuration

All configuration is by environment variable, read once at startup.

| Variable | Default | Description |
|----------|---------|-------------|
| `BABYLONJS_INSPECTOR_PORT` | `9070` | Port for the HTTP + WebSocket server. Must be 1-65535 or startup fails with an explicit error. |
| `BABYLONJS_INSPECTOR_HOST` | `127.0.0.1` | Interface the server binds to. Loopback by default so the bridge is not reachable from other machines. Must be non-empty. |
| `BABYLONJS_INSPECTOR_ALLOWED_ORIGINS` | (none) | Comma-separated list of extra browser origins allowed to open a WebSocket. Loopback origins are always allowed. `*` disables the check. |
| `GEMINI_API_KEY` | (none) | Gemini API key. When unset, `babylonjs_scene_analyze` is not registered. |
| `BABYLONJS_INSPECTOR_SCREENSHOT_DIR` | `./tmp/screenshots` | Directory for screenshot output. Read with a default applied, but not validated and not yet consumed by any code path. |
| `BABYLONJS_INSPECTOR_LOG_LEVEL` | `info` | One of `debug`, `info`, `warn`, `error`. Validated at startup, but no logger reads it yet. |

An invalid `BABYLONJS_INSPECTOR_PORT`, `BABYLONJS_INSPECTOR_HOST` or
`BABYLONJS_INSPECTOR_LOG_LEVEL` throws at startup rather than silently falling back to the
default.

## Security

The inspector exists to run code you asked for inside your own page, which means anything
that can reach the bridge can run `babylonjs_scene_evaluate` against your scene and feed the
answer to your LLM. Two defaults keep that surface small:

- **Loopback only.** The HTTP + WebSocket server binds `127.0.0.1`. Nothing else on your
  network can reach it unless you set `BABYLONJS_INSPECTOR_HOST` yourself.
- **Origin-checked WebSocket upgrades.** WebSockets are exempt from CORS, so the server
  checks the `Origin` header on the upgrade itself. Origins on `localhost`, `127.0.0.1` or
  `[::1]` (any port, any scheme) are accepted; every other origin is rejected with a 403
  unless you list it in `BABYLONJS_INSPECTOR_ALLOWED_ORIGINS`. Requests that send no `Origin`
  at all -- curl, a native client -- are accepted, because they can only arrive over the
  bound interface.

The HTTP routes echo `Access-Control-Allow-Origin` only for an origin that passes the same
check; there is no wildcard CORS. If a second browser connects, the previously attached
socket is closed rather than silently displaced.

## HTTP endpoints

Alongside the WebSocket upgrade handler, the server exposes:

| Endpoint | Purpose |
|----------|---------|
| `GET /health` | Returns `{"status":"ok","connected":<bool>}`. Useful for checking the server is up and whether a browser has attached. |
| `GET /inspector-client.js` | Serves the browser client bundle, with CORS headers echoed back to any origin that passes the WebSocket origin check, so a local dev server can load it. Currently a stub. |

## Implementation status

The package is built in phases. This is what actually runs today, verified against the source
rather than the design document.

**Working:**

- MCP server over stdio, with all six tools registered and their input schemas validated by zod
- Conditional registration of `babylonjs_scene_analyze` based on `GEMINI_API_KEY`
- HTTP + WebSocket server, `/health` endpoint, WebSocket upgrade handling, clean shutdown on
  SIGINT and SIGTERM
- The WebSocket wire protocol: command/response/event/ping/pong message types, strict parsing
  that rejects malformed messages, UUID request correlation, per-operation timeout tiers, and
  rejection of all pending requests when the socket closes
- Configuration loading and validation from the environment
- Graceful, readable error responses from every tool when the precondition it needs is missing

**Not yet working:**

- **The browser client is a stub.** `InspectorClient` stores its options and does nothing else;
  it does not open a WebSocket, discover a BabylonJS engine, or register command handlers.
- **`GET /inspector-client.js` serves a placeholder comment**, not a bundle. The Vite plugin
  injects a script that imports `InspectorClient` from that URL, so the injection wiring is in
  place but the import has nothing to resolve.
- Consequently every browser-dependent tool -- `scene_inspect`, `scene_evaluate`, `screenshot`,
  `spector_capture`, `scene_analyze` -- returns "No BabylonJS scene connected" in practice.
  `babylonjs_connection_status` is the only tool that returns real data today.
- **Spector.js is not integrated.** There is no browser-side capture code, so `hasSpector` is
  never set true.
- **Gemini analysis is not implemented.** `GeminiClient.analyzeScene` rejects, and the
  `babylonjs_scene_analyze` handler returns an explicit "not yet implemented" message even
  with a valid API key and a connected browser.
- **Screenshots, depth maps, wireframe and annotated views** have no capture implementation.
- **NullEngine (in-process, headless) runtime** is designed for by the `BabylonJSRuntime`
  interface but has no implementation; `BrowserBridge` is the only runtime.
- `BABYLONJS_INSPECTOR_SCREENSHOT_DIR` and `BABYLONJS_INSPECTOR_LOG_LEVEL` are read but
  unused. `BABYLONJS_INSPECTOR_SCREENSHOT_DIR` is not validated either -- any string, empty
  included, is accepted.
- **Nothing is published to npm yet**, so every `npx` and `npm install` route 404s. Build from
  a checkout.

In short: the server half of the system is real, the browser half is scaffolding. Install this
if you want to follow along or contribute; it is not yet useful for debugging a scene.

## Development

```bash
npm install          # install dependencies
npm run build        # compile TypeScript to dist/
npm run build:watch  # compile in watch mode
npm test             # run tests in watch mode
npm run test:run     # run tests once
npm run coverage     # run tests with a coverage report
npm run lint         # eslint plus a type check
npm run lint:fix     # auto-fix what eslint can
npm run typecheck    # tsc --noEmit only
npm run clean        # remove dist/ and coverage/
```

Coverage thresholds are 80% lines, 80% functions, 80% statements and 75% branches, and they
are enforced: `npm run coverage` exits non-zero below any of them, and CI gates the release
on that job. The suite currently sits around 96%.

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md).

## License

MIT -- see [LICENSE](LICENSE).