Skip to main content
Glama

m{ai}geXR-3d-mcp

The multi-framework 3D MCP server that actually works. Control live Three.js, A-Frame, Babylon.js, and React Three Fiber scenes from any MCP-capable AI — GitHub Copilot, Claude Desktop, Cursor, you name it — with in-world async chat so you can talk to the AI from inside the 3D canvas while it reshapes the world around you.

All four framework clients render matching, visually-aligned output from a single unified scene state. Same prompt, same scene, any engine.


Highlights

  • 4 production-ready 3D clients — Three.js, A-Frame (1.7.0 + bloom), Babylon.js (PBR), React Three Fiber + Zustand — all visually aligned

  • WebXR / VR support — enter immersive VR in all four clients; floating chat panel follows your gaze so you can talk to the AI from inside the headset

  • 9 AI providers out of the box — OpenAI (GPT-5.2), Anthropic (Claude Sonnet 4.6), Google Gemini 3.1 Pro, Mistral, Groq (Llama 3.3 70B), xAI Grok-4, Cohere Command R+, Together.ai, and local Ollama

  • 23 MCP tools — objects, lights, cameras, animation, environment, scene I/O, undo/redo, screenshots, and in-world chat

  • Persistent animations — animations survive page reloads; the server stores active animations in scene state and replays them when clients reconnect

  • Concurrent animations — multiple properties (position, rotation, scale) animate simultaneously on the same object across all engines

  • Per-framework system prompts — each client tells the AI how to generate geometries, materials, and lighting that look correct in that engine (adapted from the iOS maigeXR app)

  • In-world chat — press ~ to talk to the AI without leaving the 3D viewport; it reads your messages and answers in a floating overlay

  • Scene-aware AI — 20-turn conversation history + live scene state injection ensures the AI makes incremental edits, not destructive rebuilds

  • One command — pnpm dev starts the server + all four clients simultaneously; auto-opens the Three.js client in your browser

  • Hot-swappable AI provider — change provider mid-session from the client dropdown; no restart needed


Related MCP server: threejs-devtools-mcp

Quick start

Option A — npx (no clone needed)

npx maige-3d-mcp

This starts the MCP server via stdio. Point your MCP client (VS Code Copilot, Claude Desktop, Cursor) at the command npx maige-3d-mcp.

Option B — from source

1. Install

cd mcp-webgpu
pnpm install

2. Configure

cp .env.example .env

Add at least one API key. All variables:

Variable

Default

Purpose

WS_PORT

8083

WebSocket bridge port

CHAT_PROVIDER

openai

Active AI provider (openai | anthropic | google | mistral | groq | xai | cohere | together | ollama)

OPENAI_API_KEY

—

OpenAI

ANTHROPIC_API_KEY

—

Anthropic

GOOGLE_API_KEY

—

Google Gemini

MISTRAL_API_KEY

—

Mistral

GROQ_API_KEY

—

Groq

XAI_API_KEY

—

xAI / Grok

COHERE_API_KEY

—

Cohere

TOGETHER_API_KEY

—

Together.ai

OLLAMA_BASE_URL

http://localhost:11434

Local Ollama

AUTO_OPEN_BROWSER

true

Open browser on startup

DEFAULT_FRAMEWORK

threejs

Which client to auto-open (threejs | aframe | babylonjs | r3f)

Each provider also has a *_MODEL env var (e.g. OPENAI_MODEL=gpt-4.1) — see .env.example for all available models.

Two chat modes:

  • Relay mode (no key needed): the MCP host AI (Copilot, Claude Desktop) handles in-world chat by polling getPendingUserMessages.

  • Direct mode (key in .env): the server answers chat autonomously using the configured provider.

3. Build & run

pnpm build:server   # compile TypeScript once
pnpm dev            # start server + all 4 clients

Clients open at:

4. Register with VS Code Copilot

The .vscode/mcp.json is pre-configured. Reload VS Code and maige-3d-mcp appears in Copilot agent mode.

Alternatively, add to your global VS Code settings:

// .vscode/mcp.json (already included)
{
  "servers": {
    "maige-3d-mcp": {
      "type": "stdio",
      "command": "node",
      "args": ["packages/server/build/main.js"],
      "env": {
        "OPENAI_API_KEY": "${env:OPENAI_API_KEY}",
        "ANTHROPIC_API_KEY": "${env:ANTHROPIC_API_KEY}",
        "GOOGLE_API_KEY": "${env:GOOGLE_API_KEY}"
      }
    }
  }
}

Supported AI Providers (9)

Provider

Default Model

Available Models

Notes

OpenAI

gpt-5.2

gpt-5.2, gpt-5.2-pro, gpt-4.1, gpt-4.1-mini, gpt-4o, o3, o4-mini

Best general-purpose option

Anthropic

claude-sonnet-4-6

claude-opus-4-6, claude-sonnet-4-6, claude-sonnet-4-5, claude-haiku-4-5

Strong reasoning

Google Gemini

gemini-3.1-pro-preview

gemini-3.1-pro-preview, gemini-2.5-pro, gemini-2.5-flash, gemini-2.5-flash-lite

Latest flagship, multimodal

Mistral

mistral-large-latest

mistral-large, mistral-medium, mistral-small, open-mistral-nemo

Fast + capable

Groq

llama-3.3-70b-versatile

llama-3.3-70b, deepseek-r1-distill-llama-70b, llama-3.1-8b-instant, mixtral-8x7b

Blazing inference speed

xAI / Grok

grok-4-0709

grok-4-0709, grok-4-fast-reasoning, grok-3, grok-3-mini, grok-code-fast-1

Latest Grok 4

Cohere

command-r-plus

command-r-plus, command-r, command, command-light

Tool-use focused

Together.ai

DeepSeek-R1-Distill-Llama-70B-free

DeepSeek-R1-Distill-70B-free, Llama-3.3-70B-free, Llama-3.3-70B, DeepSeek-R1, Qwen2.5-72B

Free tier available

Ollama

llama3.2

llama3.2, mistral, phi4, gemma3, qwen2.5, deepseek-r1

Fully local, no API key

Switch providers from the dropdown in the chat overlay or by changing CHAT_PROVIDER in .env. Override the model per-provider with *_MODEL env vars (see .env.example).


Available MCP Tools (23)

Objects

Tool

Description

createObject

Add a mesh — box, sphere, cylinder, cone, torus, plane, capsule, or glTF model

updateObject

Partial update: position, rotation, scale, material (color, metalness, roughness, emissive), visibility

deleteObject

Remove by id

cloneObject

Duplicate with optional offset

getObject

Inspect a single object

getSceneState

Full scene JSON snapshot

Lights

createLight · updateLight · deleteLight — ambient, directional, point, spot, hemisphere

Camera

setCamera · flyToObject

Animation

animateObject · stopAnimation — rotate, bounce, pulse, float, spin, custom keyframes. Animations persist in server state and replay automatically on page reload. Multiple properties animate concurrently per object.

Environment

setEnvironment — background color, fog, tone mapping, exposure, shadow toggle

Scene

clearScene · loadScene · exportScene · undo · redo · takeScreenshot

In-world Chat

Tool

Description

getPendingUserMessages

Retrieve messages typed from inside the 3D canvas

sendChatMessage

Display AI reply in the floating overlay

clearPendingMessages

Flush the queue


MCP Resources

URI

Description

maige-3d://scene/state

Live JSON snapshot of all objects, lights, camera, and environment

maige-3d://server/sessions

List of currently connected browser sessions (id, framework, timestamp)


MCP Prompts

Prompt

Description

3d-world-assistant

Full system context for AI assistants — scene tools, chat workflow, incremental update rules, 10+ advanced demo recipes (galaxy, DNA helix, neon tunnel, crystal cluster, etc.)

framework-guide

Per-framework geometry/material/lighting tips. Accepts framework argument: threejs, aframe, babylonjs, r3f

demo-showcase

NEW: Instant access to 10+ stunning pre-built demo templates. Perfect for quick impressive visualizations. Accepts style argument: galaxy, dna, tunnel, crystals, geometries, wave, spiral, orbit, explosion, all


WebXR / VR Support

All four clients support immersive VR via WebXR. Click the 🥽 Enter VR button (bottom-left) to start a session.

Framework

Implementation

Three.js

Custom VRSetup.ts — WebXR session management, controller ray casters, VRChatPanel.ts canvas-texture chat panel

A-Frame

Native vr-mode-ui + laser-controls, 3D chat entity with dynamic text

Babylon.js

WebXRDefaultExperience + DynamicTexture chat panel

React Three Fiber

@react-three/xr v6 (createXRStore + <XR> wrapper), React VR chat panel component

In VR, the chat panel floats in front of you and follows your gaze. AI replies appear in real-time so you can direct the scene from inside the headset.

Requires a WebXR-capable browser (Chrome 79+, Edge 79+, Meta Quest Browser) and a VR headset or the WebXR API Emulator extension for desktop testing.


In-world Chat

Press ~ (backtick) or click AI Chat in the bottom-right corner. Type a message, hit Enter, and the AI receives it, acts on it, and replies — all without leaving the 3D viewport. The chat overlay also includes:

  • Provider selector — switch AI providers on the fly

  • System prompt editor — customise the AI's behaviour per session

  • Clear Scene button — reset the world instantly

  • Debug panel — press Escape to inspect scene state and connection info


Architecture

┌─────────────────────────┐
│   MCP Host (Copilot,    │  stdio / JSON-RPC
│   Claude, Cursor, etc.) │◄────────────────────┐
└─────────────────────────┘                     │
                                                 │
                              ┌──────────────────┴──────────────────┐
                              │      MCP Server (Node.js)           │
                              │                                      │
                              │  tools/ ─ 23 tool definitions        │
                              │  state/ ─ SceneStateManager + Undo   │
                              │  chat/  ─ ChatRelay (9 providers)    │
                              │  ws/    ─ WebSocket bridge :8083     │
                              │     └── adapters/ (per-framework)    │
                              └──────────┬───────────────────────────┘
                                         │ WebSocket
        ┌────────────────┬───────────────┼───────────────┬────────────────┐
        ▼                ▼               ▼               ▼                │
┌──────────────┐ ┌──────────────┐ ┌───────────────┐ ┌──────────────┐     │
│  Three.js    │ │  A-Frame     │ │  Babylon.js   │ │  R3F / React │     │
│  :5173       │ │  :5174       │ │  :5175        │ │  :5176       │     │
│ ┌──────────┐ │ │ ┌──────────┐ │ │ ┌──────────┐  │ │ ┌──────────┐ │     │
│ │VR/WebXR  │ │ │ │VR/WebXR  │ │ │ │VR/WebXR  │  │ │ │VR/WebXR  │ │     │
│ │ChatPanel │ │ │ │ChatPanel │ │ │ │ChatPanel  │  │ │ │ChatPanel │ │     │
│ └──────────┘ │ │ └──────────┘ │ │ └──────────┘  │ │ └──────────┘ │     │
└──────────────┘ └──────────────┘ └───────────────┘ └──────────────┘

Each client connects via WebSocket to the same MCP server. The server maintains a single canonical scene state and pushes commands through per-framework adapters that translate Vec3 formats, material models, and geometry names into each engine's native representation.

Key server features:

  • Conversation history — the AI remembers the last 20 turns of dialogue, avoiding destructive scene rebuilds

  • Scene state injection — every AI call includes a summary of current objects, lights, and environment so the AI knows what already exists

  • Per-framework system prompts — each client tells the AI how to generate geometries, materials, and lighting that look correct in that specific engine

  • Undo/redo — 20-deep snapshot stack on the server, triggered via MCP tools

  • Animation persistence — active animations are stored in scene state and replayed on reconnect, so looping animations survive page reloads


Project Layout

mcp-webgpu/
├── .env.example                   ← all env vars + model lists documented
├── .vscode/mcp.json               ← pre-configured for VS Code Copilot agent mode
├── package.json                   ← pnpm workspace root
├── PLAN.md                        ← full architecture plan
├── packages/
│   ├── server/                    ← MCP server (TypeScript / Node)
│   │   └── src/
│   │       ├── main.ts            ← entry + .env discovery
│   │       ├── types.ts           ← shared types
│   │       ├── tools/             ← 23 MCP tool definitions
│   │       ├── handlers/          ← tool / prompt / resource handlers
│   │       ├── state/             ← SceneStateManager + UndoStack
│   │       ├── chat/              ← ChatRelay (9 providers) + MessageQueue
│   │       └── ws/                ← WebSocket server + framework adapters
│   │           └── adapters/      ← ThreeAdapter, AFrameAdapter,
│   │                                 BabylonAdapter, R3FAdapter
│   ├── client-threejs/            ← Three.js (Vite)
│   │   └── src/
│   │       ├── scene.ts           ← SceneManager
│   │       ├── commands/          ← command dispatcher
│   │       ├── overlay/           ← ChatOverlay UI
│   │       └── vr/               ← VRSetup + VRChatPanel (WebXR)
│   ├── client-aframe/             ← A-Frame 1.7.0 + bloom (Vite)
│   │   └── src/
│   │       ├── scene.ts           ← A-Frame SceneManager
│   │       ├── commands/          ← command dispatcher
│   │       └── overlay/           ← ChatOverlay UI
│   ├── client-babylonjs/          ← Babylon.js + PBR (Vite)
│   │   └── src/
│   │       ├── scene.ts           ← Babylon SceneManager
│   │       ├── commands/          ← command dispatcher
│   │       └── overlay/           ← ChatOverlay UI
│   └── client-r3f/                ← React Three Fiber + Zustand (Vite)
│       └── src/
│           ├── App.tsx            ← React app shell
│           ├── SceneCanvas.tsx    ← R3F canvas + XR wrapper
│           ├── store/             ← Zustand scene store
│           ├── commands/          ← command dispatcher
│           ├── overlay/           ← ChatOverlay UI
│           └── vr/               ← VRChatPanel (React XR component)

Roadmap

  • Phase 1 — Three.js client + full tool set + in-world chat

  • Phase 2 — A-Frame client (1.7.0, bloom post-processing) + Babylon.js client (PBR materials)

  • Phase 3 — React Three Fiber client (Zustand state, drei helpers)

  • Phase 3.5 — 9 AI providers, per-framework system prompts, visual alignment across all 4 engines

  • Phase 4 — WebXR / VR headset support (all 4 clients + floating VR chat panel)

  • Phase 5 — VS Code MCP config, auto-open browser, conversation history + scene state awareness

  • Phase 6 — Animation persistence, concurrent multi-property animations, enhanced default lighting (ambient + hemisphere + directional with shadows)


License

MIT

Available Tools

33 tools
addBehaviorA

Attach a continuous frame-tick behavior to an object. Behaviors run every frame until removed. Types: spin (rotate continuously), bob (oscillate up/down), orbit (circle around a point), lookAt (always face camera), pulse (rhythmic scale breathing).

Inputs: id (required), objectId (required), type (required), params.

animateObjectA

Animate an object property to a target value. Supports transforms (position, rotation, scale) and material properties (emissiveIntensity, opacity, color) for pulsing glow, fading, color shifts.

Inputs: id (required), property (required), to (required), duration, +2 more.

clearPendingMessagesA

Clear all queued in-world user messages (useful after processing a batch).

Inputs: no named parameters.

clearSceneA

Remove all user-created objects from the scene and reset default lighting.

Inputs: no named parameters.

cloneObjectC

Duplicate an object with an optional position offset.

Inputs: id (required), newId, offset.

createLightC

Add a light to the scene.

Inputs: lightType (required), id, color, intensity, +7 more.

createObjectA

Add a 3D object to the live scene. Returns the assigned id. Always call getSceneState first to understand what already exists.

Inputs: type (required), id, position, rotation, +12 more.

createParticlesA

Create a particle system (stars, dust, sparks, snow, fire embers, laser trails). Uses GPU-efficient point sprites with optional drift animation and twinkle.

Inputs: count (required), spread (required), id, position, +10 more.

deleteLightC

Remove a light from the scene by id.

Inputs: id (required).

deleteObjectC

Remove a 3D object from the scene by id.

Inputs: id (required).

deleteParticlesC

Remove a particle system from the scene.

Inputs: id (required).

executeScriptA

Execute arbitrary JavaScript code in the connected browser scene context. The code runs inside an async function with access to: scene (THREE.Scene), camera, renderer, controls, and a helpers object. Return a value to send it back. Use this for advanced effects the other tools cannot achieve — custom shaders, complex geometry, procedural generation, physics, etc.

Inputs: code (required).

exportSceneA

Export the current scene as a JSON string (can be stored and re-loaded later).

Inputs: no named parameters.

exportStandaloneSceneA

Export the current scene as a standalone HTML file that plays in any browser without a server. Includes all objects, lights, materials, animations, camera, and environment. No chat UI. Saved to the scenes/ folder. Returns the file path.

Inputs: name.

flyToObjectB

Smoothly move the camera to look at an object.

Inputs: id (required), distance, duration.

getObjectC

Get properties of a single scene object by id.

Inputs: id (required).

getPendingUserMessagesA

Return any in-world chat messages the user has typed from inside the 3D environment. Call this at the start of every turn to check for user input from the canvas.

Inputs: no named parameters.

getSceneStateA

Get the full current scene state: all objects, lights, camera, and environment. Call this first before making any changes.

Inputs: no named parameters.

listScenesB

List all saved scene JSON files in the scenes/ folder.

Inputs: no named parameters.

loadSceneB

Replace the entire scene with a previously exported scene JSON.

Inputs: sceneJson (required).

loadSceneFromFileC

Load a previously saved scene from the scenes/ folder by name.

Inputs: name (required).

redoB

Redo a previously undone scene change.

Inputs: no named parameters.

removeBehaviorC

Remove a behavior by its ID, stopping its frame-tick effect.

Inputs: id (required).

saveSceneB

Save the current scene to a JSON file in the scenes/ folder. Returns the file path.

Inputs: name.

sendChatMessageA

Display a message in the in-world chat overlay visible to the user inside the 3D view.

Inputs: message (required), sessionId.

setCameraC

Set the camera position and look-at target.

Inputs: position, target, fov.

setEnvironmentC

Configure the scene environment: background, fog, tone mapping, shadows, and post-processing effects.

Inputs: background, hdriUrl, fog, shadows, +5 more.

stopAnimationC

Stop any running animation on an object.

Inputs: id (required).

takeScreenshotA

Capture the current 3D viewport and return a base-64 PNG data URL. Requires a browser client to be connected.

Inputs: no named parameters.

undoB

Undo the last scene change (up to 20 levels).

Inputs: no named parameters.

updateLightC

Update an existing light by id.

Inputs: id (required), color, intensity, position, +1 more.

updateObjectC

Update any properties of an existing 3D object. Supports partial updates.

Inputs: id (required), position, rotation, scale, +2 more.

updateParticlesC

Update properties of an existing particle system.

Inputs: id (required), position, color, size, +4 more.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.5/5.0

    Scored across 33 tools

    Disambiguation4/5

    Most tools have distinct purposes with clear boundaries, such as createObject vs. updateObject, or animateObject vs. addBehavior. However, some overlap exists: animateObject and addBehavior both handle object animations, which could cause confusion in selection, though their descriptions clarify animateObject is for property transitions while addBehavior is for continuous frame-tick effects.

    Naming Consistency5/5

    Tool names follow a highly consistent verb_noun pattern throughout, such as createObject, deleteObject, updateObject, and getSceneState. All tools use camelCase uniformly, with no mixing of naming conventions, making the set predictable and easy to navigate.

    Tool Count3/5

    With 33 tools, the count is borderline high for a 3D scene management server, potentially overwhelming for agents. While the tools cover a comprehensive range of operations, the number may feel heavy compared to typical well-scoped servers, though it aligns with the domain's complexity.

    Completeness5/5

    The tool set provides complete coverage for 3D scene management, including CRUD operations for objects, lights, and particles, scene state management, animation controls, user interaction handling, and advanced features like scripting and export. No obvious gaps exist; agents can perform full lifecycle workflows without dead ends.

    Maintenance

    ActivityInactive
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      A
      maintenance
      Enables AI agents to inspect and modify live Three.js scenes in real time through 59 tools covering objects, materials, shaders, textures, animations, performance monitoring, memory diagnostics, and code generation. Works with vanilla Three.js, React Three Fiber, and any framework without changing your project, using an injected browser bridge and in-page devtools overlay.
      5,671 npm
      MIT
    • A
      license
      Not graded
      quality
      A
      maintenance
      Enables AI agents to inspect and edit Three.js scenes, materials, shaders, lights, animations, and performance in real time, preview changes live in the browser, capture frames, and turn approved changes into code. It supports vanilla Three.js and React Three Fiber across WebGL/WebGPU without modifying the project.
      5,671 npm
      MIT
    • A
      license
      Not graded
      quality
      A
      maintenance
      Enables AI agents to inspect and edit live Three.js scenes in the browser — objects, materials, shaders, textures, lights, animation, performance and memory — with 69 tools covering scene trees, screenshots, website audits and video frame capture. Changes can be previewed live with zero project modifications, then converted into ready-to-paste vanilla Three.js or React Three Fiber code.
      5,671 npm
      MIT
    • A
      license
      Not graded
      quality
      A
      maintenance
      Enables AI agents to inspect, debug and edit Three.js scenes in real time from a live browser, covering objects, materials, shaders, textures, lights, animation, and performance. Live preview changes can be audited, reverted, or turned into ready-to-paste vanilla Three.js and React Three Fiber code without modifying the project.
      5,671 npm
      MIT