Skip to main content
Glama
mirage-gaussifier-mcp

mirage-gaussifier-mcp

Official
by Gaussifier
README.md
# mirage-gaussifier-mcp

An [MCP](https://modelcontextprotocol.io) server that turns a photo into an editable image made
of Gaussians, and lets an assistant edit it: swirl, magnify, recolour, duplicate, tilt parts out
of the picture into 3D, and record the whole editing process as a GIF.

- [Gaussifier](https://github.com/Gaussifier/gaussifier) places the initial Gaussians in a
  fraction of a second.
- [MiraGe](https://github.com/waczjoan/MiraGe) (ICML 2025) fits flat 3D Gaussians on the image
  plane. Each one is controlled by a triangle, so any edit is a deformation of those triangles
  and the picture follows.

![editing a lighthouse photo through the MCP server](docs/demo/lighthouse.gif)

A quiet afternoon at the lighthouse gets a sunset, blue roofs and a whirlpool; then the
lighthouse steps forward, gets a free twin, and grows a big glowing head to stay the taller one.
Every step is one MCP tool call; [how it was edited](docs/demo/lighthouse_steps.md) lists them,
and [the plan](docs/demo/lighthouse_plan.json) replays it.

## How it works

1. **Gaussify.** Gaussifier's equal-mass placement of the image becomes MiraGe's initial point
   cloud on the image plane, and MiraGe fits it with a mirror camera (about 15 s for 3000
   iterations of a 768×512 image on an RTX 5090). Starting from Gaussifier gives a compact
   model: on Kodak 23 at 3000 iterations it ends with 81k Gaussians (40.9 dB), against 172k
   (47.2 dB) from MiraGe's default random start.
2. **Edit.** Each Gaussian is a triangle (its centre and two scaled axes). Selections pick
   Gaussians by region or colour; edits move the triangle vertices (or change colour and
   opacity), and the renderer rebuilds every Gaussian from its triangle. Edits stack, can be
   undone, and each one takes a strength from 0 to 1, so it can be animated.
3. **Record.** While recording, every edit is captured as a short eased animation with a
   caption. The GIF opens with the Gaussians growing from dots into the photo and can end with
   a 3D turn of the result.

## Installation

Requirements: Linux, an NVIDIA GPU, CUDA 12.8 or newer with `nvcc`, CMake, git, curl and
[uv](https://docs.astral.sh/uv/).

```bash
git clone <this repository> mirage-gaussifier-mcp
cd mirage-gaussifier-mcp
bash scripts/setup.sh
```

`scripts/setup.sh` fetches MiraGe (pinned commit, with two small compatibility fixes for current
compilers and Pillow) and Gaussifier v0.12.0 with its released weights into `third_party/`,
builds the CUDA extensions and Gaussifier's runtime for the local GPU, and creates the
environment.

## Use it from an MCP client

The server speaks MCP over stdio:

```json
{
  "mcpServers": {
    "mirage-gaussifier": {
      "command": "uv",
      "args": ["run", "--directory", "/path/to/mirage-gaussifier-mcp", "mirage-gaussifier-mcp"]
    }
  }
}
```

Put this in your client's MCP configuration (for example `.mcp.json` in a project, or the
desktop app's settings). Then ask, for example: *"Gaussify kodim23.png, make the left parrot
green, swirl the right one, and record it as a GIF."*

To script it without a client, `scripts/mcp_session.py` runs a JSON list of tool calls in one
session and saves every preview:

```bash
uv run python scripts/mcp_session.py docs/demo/<name>_plan.json --out previews/
```

## Tools

| Tool | What it does |
|---|---|
| `gaussify_image` | Image → editable Gaussian image (`init="gaussifier"` or `"random"`, `iterations`, `max_side`). Reuses an existing fit unless `refit`. |
| `render` | Current result, optionally with the image plane turned in 3D (`yaw`, `pitch`). |
| `select_region` | Name a set of Gaussians: `box`, `ellipse`, `polygon` (fractions of the image), `color` (`rgb`, `tolerance`), `mask_image`, `all`; combine with `add` / `subtract` / `intersect`. Returns the count and a highlighted preview. |
| `edit` | Apply one edit to a selection (below). |
| `list_operations` | The edits and their parameters. |
| `undo`, `reset`, `history` | Edit stack. |
| `start_recording`, `stop_recording` | Record the editing process as a GIF (intro, captions, 3D finale). |
| `animate_edit`, `orbit` | Loop one edit, or turn the result in 3D, as a GIF. |
| `export_pseudomesh`, `import_pseudomesh` | The triangles as an OBJ, to deform in Blender and bring back (MiraGe's workflow). |

Edits: `translate`, `rotate`, `scale`, `flip`, `bulge` (magnify or pinch), `swirl`, `wave`,
`tilt3d` (turn a part out of the image plane), `lift` (toward the viewer), `recolor` (hue,
saturation, brightness, tint), `fade`, `duplicate`. Positions and sizes are fractions of the
image.

Limits: moving or shrinking a region leaves an empty (black) area behind it, since nothing is
inpainted; region selections include the background inside them, so intersect a region with a
colour selection (or use a polygon) to isolate an object.

## Licences

This repository's code is MIT licensed. MiraGe, which `scripts/setup.sh` downloads and this
server imports, is distributed under its authors' Gaussian-Splatting licence for
non-commercial research and evaluation use only (see `third_party/MiraGe/LICENSE.md`), and so
is any use of this server with it. Gaussifier is used through its public release.