Shorts Pipeline MCP Server

A remote MCP server that wraps the xAI Imagine API so Claude (web and mobile) can generate dark editorial-cartoon animated vertical shorts through tool calls.

It ships with a second control surface: Shorts Studio, a web UI at /studio that drives the same pipeline directly — see Web studio.

Setup

1. Supabase

Create a project, then run supabase/schema.sql in the SQL editor. It creates projects, shots, assets, jobs, story_manifests and reference_assets, and is safe to re-run.

Upgrading an existing database: re-run the same file. It now adds a critique column to assets (add column if not exists), which is where the vision pass's verdict is stored so the studio can show why a variation was rejected. Everything keeps working without it — the critique panel just stays empty.

The storage bucket (shorts, public) is created automatically on first boot — no manual step needed.

You need the service role key, not the anon key: the server writes to Storage and bypasses RLS. It is the only thing holding the database, and it never leaves the process.

2. Environment

cp .env.example .env

Fill in XAI_API_KEY, SUPABASE_URL, SUPABASE_SERVICE_KEY, and generate a shared secret:

node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"

3. Run

npm install
npm run dev      # tsx watch
# or
npm run build && npm start

ffmpeg is bundled via ffmpeg-static — no system install required. The server verifies it at boot and refuses to start without it, since frame extraction would silently fail otherwise.

4. Expose it

Claude reaches the server from Anthropic's infrastructure and cannot reach localhost. You need a public HTTPS URL.

cloudflared tunnel --url http://localhost:3000

Free-tier tunnel URLs change on every restart and the connector has to be re-added each time. If that loop gets tedious, deploy early and iterate against the deployed instance.

Production: Render (free). render.yaml and Dockerfile are ready to go.

1. Push this repo to GitHub.

2. Render → New → Blueprint → select the repo. It reads render.yaml.

3. Render prompts for the four secrets (XAI_API_KEY, SUPABASE_URL, SUPABASE_SERVICE_KEY, SHORTS_SHARED_SECRET) — they are marked sync: false so they never enter the repo. Copy them from your local .env.

4. Deploy. Your URL is https://<service-name>.onrender.com.

Note that Cloudflare Workers is not an option: frame extraction shells out to ffmpeg, which Workers cannot run. The image installs ffmpeg from apt rather than via ffmpeg-static; src/frames.ts picks whichever is present and honours FFMPEG_PATH.

The free instance sleeps after 15 minutes idle and takes roughly a minute to wake. The first tool call in a chat after an idle period may therefore be slow enough that Claude reports a connection error — call it again and it will work. Two ways to avoid it:

• Ping GET /healthz every ~10 minutes from an external uptime monitor. Free instances get 750 hours/month and a full month is ~744, so staying awake around the clock just fits.

• Upgrade to a paid instance.

Sleeping is survivable rather than harmful because check_job polls upstream eagerly and all job state is in Postgres — a job in flight when the instance sleeps resumes on the next call. The one behavioural difference from an always-on host: a job is not finished unattended, so if you call animate and abandon the chat, it completes the next time anything calls check_job.

Health probe for the host: GET /healthz (unauthenticated).

5. Add the connector in Claude

Settings → Connectors → Add custom connector (Pro/Max/Team/Enterprise).

URL: your HTTPS URL with the shared secret as the path segment:

https://<your-host>/<SHORTS_SHARED_SECRET>

Leave Advanced settings (OAuth Client ID/Secret) blank. Those fields are for OAuth client registration; the shared secret does not go there.

Why the secret is in the URL: claude.ai's request-header support is a gated beta ("contact Anthropic for early access"), so on most accounts the connector dialog offers nothing but OAuth fields — there is no place to type a key. A capability URL is the remaining option, and over HTTPS the path is inside the TLS session, making it about as strong as a bearer token.

The tradeoff is that URLs get logged more casually than headers: your host's access logs will contain the secret. Our own logger redacts it, but treat the whole URL as a credential, and rotate SHORTS_SHARED_SECRET if it leaks.

If you do have the header beta, or you're connecting from Claude Code, point at the bare root URL instead and send the secret as a header. Only allowlisted names reach the server — authorization, x-api-key, x-auth-token — and all three are accepted (authorization with or without a Bearer prefix). A bespoke name like x-shorts-key will never arrive, which is why AUTH_HEADER defaults to x-api-key.

Add it on the web first. iOS and Android can use remote MCP servers but cannot add new ones.

Related MCP server: Sora MCP Server

Web studio

Open https://<host>/studio in a browser. It asks once for SHORTS_SHARED_SECRET, keeps it in localStorage, and sends it as x-api-key on every request — the same credential path Claude Code uses.

It is the same pipeline, not a viewer. Everything it does goes through src/pipeline.ts, which is also what the MCP tools call, so a short started in Claude can be finished in the browser and vice versa.

The wizard's first four steps call Grok (/api/assist → chat/completions) to pitch hooks, draft the ~40-second story, break it into 5-second chunks, and work out the minimum reference plates. Step 5 turns the plan into real rows: it creates the project, saves the manifest, generates the plates, then generates stills shot by shot with a visible log. Video is deliberately not submitted — you approve plates first, then press Animate approved.

Things worth knowing

• The studio spends money. Generating stills and submitting video are real xAI calls. The shared secret in front of /api is protecting a wallet, not just data. /studio itself is unauthenticated because it is a static document with nothing in it.

• Cancel is local. xAI exposes no cancel endpoint, so cancelling stops the worker polling and frees the shot; the upstream generation still completes and is still billed.

• Rough cut is not a render. It plays the separate clips in sequence. There is no concatenated file on the server, and nothing pretends there is. Downloading gives you the individual mp4s, named <project>-shot-01.mp4 and zero-padded so they sort into story order on a timeline. Saving several at once makes the browser ask permission once — that is the browser working correctly, not an error.

• Downloads proxy through the server. The clips already have public Storage URLs, but a browser ignores the download attribute on a cross-origin link and plays the mp4 in a tab instead of saving it. GET /api/shots/:id/video streams it back from our own origin with a real filename.

• Reference plates create shots. Reference assets hang off a shot row, so importing one makes a shot. The studio marks these and keeps them out of the Shots tab and the shot counts.

Rebuilding the front end

The studio is bundled to a single self-contained document and committed as src/ui.generated.ts, for the same reason the widgets are: the Dockerfile runs tsc only, and esbuild's platform binary is not reliably present under npm ci --ignore-scripts.

npm run build:ui        # ui/*.ts -> src/ui.generated.ts
npm run typecheck       # server + browser sources

Commit src/ui.generated.ts with any change under ui/, or the deployed studio will be stale.

Tools

Projects are addressed by project_name on generate_still (created on first use) and import_image; list_shots returns every project, so multiple shorts stay separate without adding create_project/list_projects tools.

External images

import_image is the bridge for hosts that generate an image outside this MCP server, then want Grok to animate it. Pass exactly one of image_url, image_base64, or image_file.data; the server validates PNG/JPEG/WebP bytes (20MB max), saves them to Storage, creates a normal still asset, and returns an asset_id:

import_image(image_url = "https://example.com/generated.png", project_name = "My Short")
approve_still(asset_id = "<returned asset_id>")
animate(asset_id = "<returned asset_id>", motion_instruction = "slow push in")

If ChatGPT regenerates a faulty scene image, pass the original shot_id to import_image. The imported file is stored as another still under that same shot; approve_still on the returned asset_id makes the replacement the current source for animate.

This keeps animate unchanged: it still only accepts an approved still asset_id, but that still no longer has to be generated by generate_still.

Story memory and references

For a full short, have the host analyze the story first, save a compact manifest, then import reusable reference plates:

save_story_manifest(project_name = "My Short", manifest = { ... })
import_reference_image(role = "character", label = "Mara main design", image_url = "...")
import_reference_image(role = "location", label = "Market street", image_url = "...")
get_story_manifest(project_name = "My Short")
generate_still(
  shot_description = "Mara crosses the market at dawn...",
  reference_asset_ids = ["<Mara asset_id>", "<Market street asset_id>"]
)

The manifest lives in Postgres and reference images are normal stored assets with labels, roles and URLs, so a new chat can resume the story without relying on conversation memory.

Reference images (character continuity)

generate_still accepts up to three reference_asset_ids. Pass the approved still that established a character and the new shot keeps their face, costume and proportions instead of being an independent roll of the dice:

generate_still(
  shot_description = "The same man now on a narrow cobbled street at night...",
  reference_asset_ids = ["<asset_id of shot 1's approved still>"]
)

References resolve to assets in your own Storage, never an expiring upstream URL, and any still or extracted video frame can serve as one (a video asset cannot — use its first_frame/last_frame). When references are present the request routes to /v1/images/edits instead of /v1/images/generations, and a server-owned REFERENCE_BLOCK is appended instructing the model to hold the design — continuity is not something the caller should have to remember to ask for, for the same reason style isn't.

Request shapes differ by count and both were confirmed against the live API: one reference goes in image as an object, several go in images as an array. Passing an array to image is rejected with 422.

Typical flow:

generate_still -> approve_still -> animate -> check_job (poll)

Current ChatGPT story flow does not run a first/last-frame approval pass. Submit all scene videos first, let the player widgets show them, then have the user name any bad scene videos to regenerate.

Design notes

Style is server-owned

STYLE_BLOCK and NEGATIVE_BLOCK live in src/config.ts and are concatenated onto every prompt automatically:

`${shot_description} ${motion_instruction} ${STYLE_BLOCK} ${palette_override} ${NEGATIVE_BLOCK}`

No tool exposes a style parameter, and the smoke test asserts that. The xAI endpoints are stateless — nothing carries between calls — and in manual testing omitting the style block produced fully photorealistic output while the model repeatedly tried to make characters face camera and speak. Making style a server-owned constant means neither the user nor Claude can forget it.

palette_override is the one prompt lever exposed, appended after the style block, because palette shifts per story beat while the base style never does.

Every asset is re-hosted immediately

Upstream URLs are treated as ephemeral. Every generated asset is downloaded and written to Supabase Storage before the tool returns, and only our own URLs are handed back. The previous pipeline (Higgsfield) failed mid-project when upstream media IDs expired and the style reference plates became unreachable.

Last frames are stored automatically

Finished videos still have extracted frame assets in Storage. The normal check_job result returns the video_url and last_frame_url; the last frame is useful later as a continuity reference or a handoff point.

check_job no longer embeds a first/last-frame comparison by default. Pass include_images: true, include_frames: true, or critique: true only when explicitly troubleshooting a bad clip.

generate_still returns one contact sheet, not one image per variation. Four variations tile into a single image (597kB of PNGs → a 69kB JPEG in ~130ms), so a call costs one image slot instead of four. That matters because hosts may cap how many images a conversation will carry, and four blocks per call exhausts such a budget quickly — after which later images arrive as empty slots. Side by side is also the better way to compare variations. Pass image_mode: "individual" for one block each.

Inline UI — MCP Apps (SEP-1865)

Image content blocks go to the model; they put nothing on the chat surface for the human. So generate_still and check_job also declare UI resources, which the host renders in a sandboxed iframe inline in the conversation:

The player is attached to animate as well as check_job, so a card appears the moment a job is submitted rather than only once someone remembers to poll. While the job runs the widget polls check_job from inside the iframe and updates in place — elapsed time, status, then the video. Its own polls pass include_images: false and critique: false, since the widget only needs URLs and re-running the vision pass every five seconds would cost real money for something nobody reads. If the host does not support proxying tool calls, the card just holds its last known status instead of erroring.

The two mechanisms are complementary and both are kept — the widget is for you, the image blocks are for the model, and neither replaces the other. This is an optional, backwards-compatible extension: hosts without support ignore the _meta and the resource and still get identical text and image content.

Only URLs cross the wire; the media loads directly from Storage. Two details that are easy to get wrong:

• Sandbox CSP is deny-by-default. Without declaring the bucket origin in _meta.ui.csp.resourceDomains, the iframe may load no images or video at all and the widgets render empty.

• Resources must be fully self-contained. The iframe cannot fetch sibling assets, so CSS and JS are inlined into a single document. widgets/build.mjs bundles widgets/*.client.ts with esbuild into src/widgets.generated.ts.

That generated file is committed deliberately: the Docker build runs tsc directly rather than npm run build, because esbuild's platform binary is not reliably present under npm ci --ignore-scripts. After editing a widget, run npm run build:widgets and commit the result.

The vision pass — judging when the images don't arrive

Image content blocks are unreliable in practice. Hosts appear to cap how many images a conversation will carry; past that limit they arrive as correctly positioned but empty slots, and the model is silently blind. Reproduced on a single connection: the same finished job returned the same bytes early in a conversation and empty slots later, with nothing changing server-side.

Text always gets through. So the server looks at the image itself and returns a structured judgement as text:

"critiques": [{
  "variation": 3, "asset_id": "e89658f3",
  "verdict": "regenerate",
  "reason": "Character faces are too realistic and handsome instead of exaggerated editorial-cartoon caricatures.",
  "style_ok": false, "palette_ok": true, "framing_ok": true,
  "people": 1, "visible_text": false, "faces_to_camera": false,
  "anatomy_issues": null,
  "fix_suggestion": "Push oversized expressive eyes, exaggerated nose and brows, angular cheeks, simplified skin planes, painterly texture, and the dark green, burnt orange, red and black palette..."
}]

check_job can return a drift_report comparing first and last frame only when called with critique: true; this is a diagnostic path, not the regular video approval flow.

The rubric is built from STYLE_BLOCK and NEGATIVE_BLOCK, so it tests the same rules the prompt asserts. Verdicts are re-derived server-side rather than trusting the model to apply its own rule: any failing flag forces regenerate, because a self-contradictory "accept" is exactly what this exists to prevent.

This is a floor, not a replacement for seeing. It is a description, and a model relying on it is judging a description — it should say so. Measured against four real stills it was discriminating (2 accept, 2 regenerate, catching style and palette violations) but not identical to a direct look. Images are still returned alongside; this only carries the load when they don't arrive.

Costs ~$0.006 and ~4–5s per image. ENABLE_VISION_CRITIQUE=false, or critique: false per call, turns it off. VISION_MODEL defaults to grok-4.5.

If images stop reaching the model: ENABLE_MCP_APPS=false

Declaring a UI resource on a tool appears to change how some hosts handle that tool's result — it gets routed to the widget, and the image content blocks may stop reaching the model. That trade is a bad one: the model comparing first and last frames is the load-bearing quality check, and a widget the human looks at does not replace it.

ENABLE_MCP_APPS=false removes both the resources and the tools' _meta, in one env var, no code change. Set it in the host dashboard and restart.

To tell server faults from host rendering, run npx tsx scripts/live-tool-check.ts — it prints the actual content blocks a deployed tool call returns and validates that each image decodes to a real JPEG. If that shows valid images, the server is fine and the problem is host-side.

Jobs survive restarts

Job state lives entirely in Postgres. The worker re-reads open jobs from the jobs table every tick and runs one pass immediately at boot, so killing the process mid-job loses nothing — which matters during development, where the tunnel drops frequently. check_job also polls upstream eagerly, so it makes progress even between worker ticks.

If post-processing fails after a successful generation (network blip, ffmpeg error), the job is left open and retried rather than failed — a paid generation is never discarded over a transient error.

Model choice

grok-imagine-video, not grok-imagine-video-1.5. The older model holds the stylized editorial-cartoon look far better; 1.5 pulls toward photorealism and costs twice as much. Default resolution is 720p (1080p does not exist on the old model; upscaling happens in post).

Both are set via VIDEO_MODEL / VIDEO_RESOLUTION env vars, so swapping models is a config change rather than a refactor.

Auth

One static shared secret, accepted two ways and compared in constant time:

• /<secret> — credential as the URL path segment, for claude.ai connectors.

• / — credential in a header (authorization, x-api-key, or x-auth-token), for Claude Code and anything else that can set one.

Full OAuth with Dynamic Client Registration is overkill for a single-user service with no per-user data, but an unauthenticated public URL lets anyone burn your xAI credits.

The spec assumed claude.ai connectors could send an arbitrary static header. They cannot: header support is a gated beta, and even with it, names are restricted to an allowlist that excludes bespoke ones. Hence the URL-path route.

Both checks are Express middleware in src/auth.ts, so swapping in OAuth later touches no tool logic.

The studio's /api mount reuses the header check unchanged. /studio itself is served without auth: it is a static document containing no project data, and it can read nothing until someone types the secret into it. The alternative — a secret in the URL — would put the credential in browser history, screenshots and whatever the browser syncs.

Secrets

The xAI key is injected in src/xai.ts and nowhere else. Rather than trusting every call site, the logger runs every line through redact() on the way out, and upstream error bodies are redacted before they can propagate into a tool response.

Transport

Streamable HTTP only, served at the root path, with explicit session management. Legacy HTTP+SSE is deliberately not implemented — it is being deprecated.

• HEAD / — protocol discovery (gated)

• POST / — JSON-RPC; initialize opens a session, id returned in mcp-session-id

• GET / — server→client notification stream

• DELETE / — ends a session

• Unknown session id → 404, telling the client to re-initialize

/healthz, /studio and /api/* are mounted before the MCP routes, because /:token matches any single path segment and would otherwise swallow them. The MCP mounts keep a 4 MB JSON limit; /api gets its own 32 MB parser, since a 20 MB reference plate is ~27 MB of base64.

Tests

npm run typecheck
npm run smoke

The smoke test boots the real Express app in-process with dummy credentials and verifies auth rejection, HEAD discovery, stale-session handling, the full Streamable HTTP handshake, the exact tool surface, prompt assembly ordering, and key redaction — without spending anything upstream.

Not covered offline (these need live credentials): actual image/video generation, Supabase persistence, and end-to-end frame extraction on real output. See the acceptance checklist in shorts-mcp-server-spec.md.

Out of scope for v1

Web UI, audio/narration, video assembly, upscaling, multi-user/billing, OAuth.

The spec also deferred the edit capability; it has since been added as reference_asset_ids on generate_still (see above), which covers both character continuity and the spec's stated motivation of fixing drift without a full re-roll.