babylonjs-inspector-mcp

An MCP (Model Context Protocol) server that lets an LLM inspect a running BabylonJS 3D scene.

Status: early development (0.x). The MCP server, the tool surface, the WebSocket protocol and the HTTP server are implemented. The browser-side client that actually talks to your BabylonJS scene is still a stub, so today the tools connect and validate but return "no scene connected" until that lands. Read Implementation status for the exact breakdown before you install this.

The problem

When an LLM writes BabylonJS code -- shaders, geometry, materials, lighting, animations, post-processing -- it has no way to check the result. It cannot:

• See what the scene looks like, from any angle

• Tell whether two objects are actually where it intended them to be

• Read back shader compilation results or WebGL state

• Confirm that a material or light produces the effect it was asked for

• Debug a rendering artifact it cannot observe

The LLM writes code, the human looks at the screen, and the human becomes the LLM's eyes.

Related MCP server: threejs-devtools-mcp

The solution

An MCP server that gives the LLM three kinds of access to a live scene:

1. Scene data -- structured JSON from BabylonJS: scene graph, meshes, materials, cameras, lights, animations, textures, physics, post-processing, performance counters.

2. WebGL introspection -- Spector.js frame captures: draw calls, shader source, WebGL state, texture bindings.

3. VLM scene understanding -- Gemini vision analysis of screenshots, depth maps and multi-view renders, combined with scene metadata (bring your own API key).

Plus a scene_evaluate escape hatch that runs arbitrary JavaScript against the scene, in the spirit of Playwright's browser_evaluate.

Architecture

The server is a Node.js process started by your MCP host. It speaks MCP over stdio to the host and WebSocket to your browser. It does not embed a browser; your app connects to it.

  +---------------------------+
  | MCP client                |   Claude Code, Claude Desktop, or any MCP host
  +---------------------------+
              ^
              |  stdio (JSON-RPC) -- MCP tool calls and results
              v
  +---------------------------+
  | babylonjs-inspector-mcp   |   Node.js process spawned by the MCP host
  |                           |   - registers the MCP tools
  |                           |   - BrowserBridge: correlates requests by UUID
  |                           |   - HTTP + WebSocket server on port 9070
  +---------------------------+
              ^
              |  WebSocket (ws://localhost:9070) -- commands and responses
              v
  +---------------------------+
  | Browser                   |   your Vite dev server, or any page
  |   inspector client        |   - discovers the BabylonJS engine
  |   + your BabylonJS scene  |   - runs commands against the live scene
  +---------------------------+

A tool call travels the whole chain: the host calls babylonjs_scene_inspect, the server dispatches it to the BrowserBridge, the bridge sends a WebSocket command tagged with a UUID, the browser client executes it against the scene, and the response comes back up the chain.

Only one browser connection is tracked at a time. Every operation has a timeout tier:

Requirements

• Node.js >= 20

• A BabylonJS application you can run locally

• Optionally, a Gemini API key for the VLM analysis tool

Installation

npx -y babylonjs-inspector-mcp                  # run on demand
npm install -g babylonjs-inspector-mcp          # binary on your PATH
npm install --save-dev babylonjs-inspector-mcp  # Vite plugin / browser client

To work from a checkout instead:

git clone https://github.com/apowers313/babylonjs-inspector-mcp.git
cd babylonjs-inspector-mcp
npm install
npm run build

That produces dist/, which bin/babylonjs-inspector-mcp.js needs in order to run.

MCP client configuration

Add the server to your MCP settings, pointing at your local build. For Claude Code:

claude mcp add babylonjs-inspector -- node /path/to/babylonjs-inspector-mcp/bin/babylonjs-inspector-mcp.js

Or configure it by hand -- this is the JSON block for claude_desktop_config.json, .mcp.json, or your host's equivalent:

{
  "mcpServers": {
    "babylonjs-inspector": {
      "command": "node",
      "args": ["/path/to/babylonjs-inspector-mcp/bin/babylonjs-inspector-mcp.js"],
      "env": {
        "BABYLONJS_INSPECTOR_PORT": "9070",
        "GEMINI_API_KEY": "your-key-here"
      }
    }
  }
}

After the package is published, "command": "npx" with "args": ["-y", "babylonjs-inspector-mcp"] replaces those two lines, and a global install lets you use "command": "babylonjs-inspector-mcp" with no args. Use npx -y so that npx's install prompt can never end up on the stdio channel the MCP protocol is using.

GEMINI_API_KEY is optional. When it is absent, the babylonjs_scene_analyze tool is not registered at all, so it never shows up in the LLM's tool list. When it is present, the tool appears. Everything else works without it.

Browser-side setup

The server needs your BabylonJS page to connect back to it over WebSocket. There are two supported ways to make that happen.

Vite plugin (zero instrumentation)

Add the plugin to your app's vite.config.ts. It only applies in dev mode (apply: "serve"), so production builds are untouched, and it needs no changes to your application code.

import { defineConfig } from "vite";
import { babylonjsInspector } from "babylonjs-inspector-mcp/vite";

export default defineConfig({
  plugins: [
    babylonjsInspector({
      port: 9070,        // must match BABYLONJS_INSPECTOR_PORT; default 9070
      autoConnect: true, // connect on page load; default true
    }),
  ],
});

The plugin appends a <script type="module"> tag to your HTML that imports the inspector client from http://localhost:<port>/inspector-client.js, which the MCP server serves, and assigns the instance to window.__babylonjsInspector.

Manual import

If you are not on Vite, or you want explicit control over when the inspector attaches:

import { InspectorClient } from "babylonjs-inspector-mcp/client";

const inspector = new InspectorClient({
  port: 9070,         // default 9070
  autoDiscover: true, // find the BabylonJS engine automatically; default true
  scene: myScene,     // optional: pass the scene explicitly instead
});

Both paths are wired up but not yet functional -- see below.

MCP tools

Six tools are registered. Five are always present; babylonjs_scene_analyze is registered only when GEMINI_API_KEY is set.

Every browser-dependent tool degrades to a plain JSON { "error": "..." } message rather than throwing, so the LLM gets a readable reason instead of a protocol failure.

babylonjs_connection_status

No parameters. Returns connected, engineType, canRender, hasSpector, canvasSize, and an availableTools object reporting which of sceneInspect, sceneEvaluate, screenshot, spectorCapture and sceneAnalyze are usable right now. Every entry is false while no browser is attached, because every one of those tools answers "No BabylonJS scene connected" in that state. Call this first when a tool reports that nothing is connected.

babylonjs_scene_inspect

category accepts: summary, full, meshes, mesh_detail, materials, material_detail, cameras, lights, animations, textures, physics, post_processing, performance, scene_tree, webgl_errors.

summary and full return an overview of the whole scene; every other category drills into one aspect. This is deliberately one parameterized tool rather than fifteen separate ones, to keep the host's tool list short and the LLM's choice simple.

babylonjs_scene_evaluate

Example argument:

(scene) => scene.meshes.map((m) => ({ name: m.name, pos: m.position.asArray() }))

The return value is serialized to JSON. This is a development tool with no sandbox -- the code runs with full access to the page.

babylonjs_screenshot

views accepts: current, front, top, side, isometric, depth, wireframe, annotated. customCamera.projection is perspective or orthographic.

Images are written to disk and the tool returns file paths, not base64, so a multi-view capture does not flood the context window.

babylonjs_spector_capture

summary returns counts and timings, shaders adds deduplicated shader source, full adds per-draw-call WebGL state.

babylonjs_scene_analyze

Registered only when GEMINI_API_KEY is set.

views accepts: current, front, top, side, isometric, depth, annotated.

Good questions are concrete: "Are the shadows rendering correctly?", "Is the sphere positioned above the plane?", "Does the PBR material look metallic?"

Configuration

All configuration is by environment variable, read once at startup.

An invalid BABYLONJS_INSPECTOR_PORT, BABYLONJS_INSPECTOR_HOST or BABYLONJS_INSPECTOR_LOG_LEVEL throws at startup rather than silently falling back to the default.

Security

The inspector exists to run code you asked for inside your own page, which means anything that can reach the bridge can run babylonjs_scene_evaluate against your scene and feed the answer to your LLM. Two defaults keep that surface small:

• Loopback only. The HTTP + WebSocket server binds 127.0.0.1. Nothing else on your network can reach it unless you set BABYLONJS_INSPECTOR_HOST yourself.

• Origin-checked WebSocket upgrades. WebSockets are exempt from CORS, so the server checks the Origin header on the upgrade itself. Origins on localhost, 127.0.0.1 or [::1] (any port, any scheme) are accepted; every other origin is rejected with a 403 unless you list it in BABYLONJS_INSPECTOR_ALLOWED_ORIGINS. Requests that send no Origin at all -- curl, a native client -- are accepted, because they can only arrive over the bound interface.

The HTTP routes echo Access-Control-Allow-Origin only for an origin that passes the same check; there is no wildcard CORS. If a second browser connects, the previously attached socket is closed rather than silently displaced.

HTTP endpoints

Alongside the WebSocket upgrade handler, the server exposes:

Implementation status

The package is built in phases. This is what actually runs today, verified against the source rather than the design document.

Working:

• MCP server over stdio, with all six tools registered and their input schemas validated by zod

• Conditional registration of babylonjs_scene_analyze based on GEMINI_API_KEY

• HTTP + WebSocket server, /health endpoint, WebSocket upgrade handling, clean shutdown on SIGINT and SIGTERM

• The WebSocket wire protocol: command/response/event/ping/pong message types, strict parsing that rejects malformed messages, UUID request correlation, per-operation timeout tiers, and rejection of all pending requests when the socket closes

• Configuration loading and validation from the environment

• Graceful, readable error responses from every tool when the precondition it needs is missing

Not yet working:

• The browser client is a stub. InspectorClient stores its options and does nothing else; it does not open a WebSocket, discover a BabylonJS engine, or register command handlers.

• GET /inspector-client.js serves a placeholder comment, not a bundle. The Vite plugin injects a script that imports InspectorClient from that URL, so the injection wiring is in place but the import has nothing to resolve.

• Consequently every browser-dependent tool -- scene_inspect, scene_evaluate, screenshot, spector_capture, scene_analyze -- returns "No BabylonJS scene connected" in practice. babylonjs_connection_status is the only tool that returns real data today.

• Spector.js is not integrated. There is no browser-side capture code, so hasSpector is never set true.

• Gemini analysis is not implemented. GeminiClient.analyzeScene rejects, and the babylonjs_scene_analyze handler returns an explicit "not yet implemented" message even with a valid API key and a connected browser.

• Screenshots, depth maps, wireframe and annotated views have no capture implementation.

• NullEngine (in-process, headless) runtime is designed for by the BabylonJSRuntime interface but has no implementation; BrowserBridge is the only runtime.

• BABYLONJS_INSPECTOR_SCREENSHOT_DIR and BABYLONJS_INSPECTOR_LOG_LEVEL are read but unused. BABYLONJS_INSPECTOR_SCREENSHOT_DIR is not validated either -- any string, empty included, is accepted.

• Nothing is published to npm yet, so every npx and npm install route 404s. Build from a checkout.

In short: the server half of the system is real, the browser half is scaffolding. Install this if you want to follow along or contribute; it is not yet useful for debugging a scene.

Development

npm install          # install dependencies
npm run build        # compile TypeScript to dist/
npm run build:watch  # compile in watch mode
npm test             # run tests in watch mode
npm run test:run     # run tests once
npm run coverage     # run tests with a coverage report
npm run lint         # eslint plus a type check
npm run lint:fix     # auto-fix what eslint can
npm run typecheck    # tsc --noEmit only
npm run clean        # remove dist/ and coverage/

Coverage thresholds are 80% lines, 80% functions, 80% statements and 75% branches, and they are enforced: npm run coverage exits non-zero below any of them, and CI gates the release on that job. The suite currently sits around 96%.

Contributing

See CONTRIBUTING.md.

License

MIT -- see LICENSE.