# Scratch MCP Server

An [MCP](https://modelcontextprotocol.io) server that builds **real Scratch 3 projects** from a prompt.
Ask for a scene in plain language — "put two characters on a classroom background and make them take
turns talking" — and the assistant drives this server to produce a `.sb3` file that opens in the Scratch
editor with proper sprites, costumes, backdrops, sounds and block scripts.

Nothing is faked: the output is the same project format Scratch saves, so every script is editable in
the Scratch editor afterwards.

## What it can do

- **Backdrops** — nine built-in stage backgrounds (sky, hills, night, space, room, underwater, grid,
  blank, any flat colour), or your own image file.
- **Sprites** — built-in vector shapes, or your own art.
- **Character import** — hand it a jpg/png of a character on a flat background; it keys the background
  out, trims the margins, scales it to the stage and can build *talk frames* so a still picture reads as
  talking when the costumes alternate.
- **Scripts** — a compact JSON action DSL covering motion, looks, sound, events, control, sensing,
  variables, lists and the pen extension. It compiles to genuine Scratch blocks, hat blocks included.
- **Sounds** — seven synthesised built-ins, or your own `.wav`.
- **Speech** — `add_speech` turns a line of text into a spoken sound using the machine's own offline
  text-to-speech engine, and reports the clip length plus the exact mouth-loop count needed to lip-sync
  a talking animation to it.
- **Preview** — serves the project to TurboWarp over loopback so you can watch it run before saving.
- **GIF export** — an animated-GIF encoder, so a finished scene can be sent to someone who does not
  have Scratch.

## Install

```sh
npm install
npm run build
```

Then register it with your MCP client. For Claude Code:

```sh
claude mcp add scratch -- node "C:/path/to/Scratch-MCP-Server/dist/index.js"
```

Or in a client config file (Claude Desktop, VS Code, …):

```json
{
  "mcpServers": {
    "scratch": {
      "command": "node",
      "args": ["C:/path/to/Scratch-MCP-Server/dist/index.js"],
      "env": { "SCRATCH_MCP_OUT_DIR": "C:/Users/you/Desktop" }
    }
  }
}
```

`SCRATCH_MCP_OUT_DIR` sets where `save_project` writes when no path is given; it defaults to
`./scratch-projects` under the working directory.

## Tools

| Tool | What it does |
| --- | --- |
| `create_project` | Starts a project and sets the backdrop |
| `add_backdrop` | Adds another stage backdrop, built-in or from a file |
| `add_sprite` | Adds a sprite from a built-in costume |
| `import_character` | Turns character art into a sprite: background removed, talk frames built |
| `add_costume` | Adds another costume to a sprite |
| `add_sound` | Adds a built-in or `.wav` sound |
| `add_speech` | Synthesizes a spoken line offline and attaches it as a sound |
| `list_voices` | Lists the speech voices installed on this machine |
| `add_script` | Compiles an action list into blocks under a hat block |
| `clear_scripts` | Removes every block from a target |
| `set_variable` | Declares a variable and its starting value |
| `remove_sprite` | Deletes a sprite |
| `project_info` | Lists everything in the open project |
| `save_project` | Writes the `.sb3` |
| `preview_project` | Serves it and returns a link that runs it in the browser |
| `dsl_reference` | The full DSL: hats, actions, expressions, built-in assets |

## The script DSL

A script is a target, a hat block, and a list of actions. Every action is an object with exactly one
key; an action that takes no argument can be written as a bare string.

```json
{
  "target": "Cat",
  "when": "flag",
  "do": [
    { "say": ["Hello!", 2] },
    { "forever": [ { "move": 4 }, "bounce_on_edge", "next_costume", { "wait": 0.1 } ] }
  ]
}
```

Values can be expressions instead of literals, using the same one-key rule:

```json
{ "if": { "test": { ">": [ { "var": "score" }, 10 ] },
          "do":   [ { "say": ["You win!", 2] }, { "stop": "all" } ] } }
```

Call `dsl_reference` for the complete list — it is written to be read by the assistant, so in practice
you just describe what you want and it writes the scripts.

## Making a still character talk

Scratch animates by swapping costumes. `import_character` builds two extra costumes that are subtly
squashed and stretched, so alternating them gives the bob of someone speaking — no speech bubbles
needed:

```json
{ "target": "Girl", "when": { "broadcast": "girl talks" },
  "do": [ { "repeat": { "times": 9, "do": [
            { "switch_costume": "Girl-talk-a" }, { "wait": 0.11 },
            { "switch_costume": "Girl-talk-b" }, { "wait": 0.13 } ] } },
          { "switch_costume": "Girl" },
          { "broadcast": "boy talks" } ] }
```

Two characters passing broadcasts back and forth reads as a conversation, and it loops forever.
`examples/students-talking.mjs` builds exactly that scene:

```sh
node examples/students-talking.mjs "C:/path/to/art-folder" out/students-talking.sb3
```

## Speech and lip-sync

`add_speech` uses whatever offline engine the host already has — Windows SAPI and the WinRT speech API,
macOS `say`, or `espeak-ng` on Linux. No network, no API key, no audio file to manage.

Which voices exist differs per machine, so ask `list_voices` first. Request a voice by gender or by part
of its name; a voice that is not installed is an error rather than a silent substitution:

```json
{ "target": "Boy", "text": "Almost. I still need to draw the diagram.", "voice": "male" }
```

On Windows this matters more than it looks. `System.Speech` only sees the old SAPI5 voice store, which
on many machines contains female voices only; the male voices often live in the OneCore store and are
reachable only through the WinRT API. Both paths are implemented, and each voice is tagged with the
engine that can actually speak it.

The timing trap is worth stating plainly: `play_sound_until_done` blocks, so a mouth loop written after
it animates *after* the audio has finished. Use the non-blocking `play_sound`, then the loop —
`add_speech` returns the repeat count to use:

```json
{ "target": "Boy", "when": { "broadcast": "boy talks" }, "do": [
  { "play_sound": "boy-line-1" },
  { "repeat": { "times": 13, "do": [
    { "switch_costume": "Boy-talk-a" }, { "wait": 0.11 },
    { "switch_costume": "Boy-talk-b" }, { "wait": 0.13 } ] } },
  { "switch_costume": "Boy" },
  { "broadcast": "girl talks" } ] }
```

`examples/students-talking-voice.mjs` builds the full voiced conversation:

```sh
node examples/students-talking-voice.mjs "C:/path/to/art-folder" out/students-talking-voice.sb3
```

## Opening the result

- **Scratch 3**: File ▸ Load from your computer ▸ pick the `.sb3`.
- **TurboWarp**: drag the file onto the page, or use `preview_project` for a link that loads it
  straight from localhost.

## Tests

```sh
npm test
```

`npm test` runs two suites.

`test/smoke.mjs` drives the built server over stdio exactly like an MCP client: it builds a project that
exercises motion, looks, sound, speech, variables, conditionals and key/click hats, checks that bad
input comes back as a readable error rather than a crash, and — when the sample art is present — builds
the two-character talking scene.

`test/gif-roundtrip.mjs` encodes synthetic frames and decodes them again with an independent decoder
written for the test, comparing pixels. This matters more than it sounds: a browser's `<img>` tag will
happily display a GIF whose LZW stream is subtly malformed, so a strict decode is the only check that
proves the encoder is correct.

## Notes and limits

- Sounds import from `.wav` only. The built-in sounds and all generated speech are produced locally, so
  the server never needs the network.
- Speech quality is whatever the OS ships. The voices are serviceable rather than lifelike; this trades
  realism for working offline with no key and no upload.
- Images import from `svg`, `png` and `jpg`. Background removal flood-fills inward from the border, so
  white clothing in the middle of a character survives; it expects a reasonably flat background.
- Custom blocks ("My Blocks") and the music/video/translate extensions are not implemented yet. The pen
  extension is.
- The server holds one project in memory at a time; `create_project` replaces it.

## Licence

MIT — see [LICENSE](LICENSE).