ShotFlow

English | 中文 | 日本語

Flow-file-driven AIGC orchestration platform. External agents read SOP definitions and call vendor-agnostic generation tools — no hardcoded brain, full reproducibility.

Keywords: AI video generation, text-to-video, AIGC, AI orchestration, cinematic AI, FFmpeg, MCP, FastAPI, React, edge-tts, Real-ESRGAN, RIFE, GPT-SoVITS, FunASR, voice cloning, text-to-speech, AI filmmaking, automated video production

Table of Contents

• What is ShotFlow

• Architecture Overview

• Features

• Supported Providers

• Quick Start

• Production Workflows

• MCP Tool Reference

• Agent Integration

• Project Structure

• FAQ

• Contributing

• License

Related MCP server: universal-image-mcp

What is ShotFlow

ShotFlow is an AIGC (AI-Generated Content) orchestration platform built on a single principle: separate the what from the how.

Rather than embedding generation logic in a monolithic pipeline, ShotFlow provides three components:

1. SOP flow files — Markdown documents that define, step by step, how to produce a given output type (video, image set, comic, micro-movie, visual novel).

2. Vendor-agnostic generation tools — Exposed through both a REST API and MCP (Model Context Protocol), so any agent framework can call them.

3. 13 provider integrations — From Tencent Hunyuan to Runway, HeyGen, and NovelAI, all behind a uniform BaseProvider interface.

External agents — WorkBuddy, Tencent Yuanqi, Alibaba Bailian, or Dify — read the SOP flow files and drive the tools. ShotFlow does not hardcode a "brain"; it supplies the tools and the instructions agents need to act.

Architecture Overview

graph TB
    subgraph External["External Agent Layer"]
        WB["🧠 WorkBuddy<br/><i>via shotflow-driver skill</i>"]
        YQ["🧠 Tencent Yuanqi<br/><i>via MCP</i>"]
        BL["🧠 Alibaba Bailian<br/><i>via REST / OpenAPI</i>"]
        DF["🧠 Dify<br/><i>via MCP</i>"]
    end

    subgraph ShotFlow["ShotFlow Platform"]
        MC["📡 MCP Server<br/>6 tools · JSON-RPC 2.0"]
        API["🌐 REST API<br/>OpenAPI 3.0"]
        OR["⚙️ Orchestrator<br/>Reads SOP · Calls providers"]
        FL["📋 SOP Flow Files<br/>make_video.sop.md<br/>make_image_set.sop.md<br/>make_comic.sop.md<br/>make_micro_movie.sop.md<br/>make_vn.sop.md<br/>make_nailong_video.sop.md"]
    end

    subgraph Providers["Vendor Provider Layer"]
        TX["Tencent<br/>Hunyuan Image/Video<br/>TTS"]
        AL["Alibaba<br/>Wanxiang (Wanx)"]
        KD["Kling · Jimeng<br/>Runway · HeyGen"]
        AU["Suno · Liblib<br/>NovelAI"]
    end

    WB -->|"MCP stdio"| MC
    YQ -->|"MCP"| MC
    BL -->|"REST"| API
    DF -->|"MCP"| MC

    MC --> OR
    API --> OR
    OR --> FL
    OR -->|"generate(kind, params)"| TX
    OR -->|"generate(kind, params)"| AL
    OR -->|"generate(kind, params)"| KD
    OR -->|"generate(kind, params)"| AU

    style ShotFlow fill:#1a1a2e,color:#e0e0e0
    style External fill:#16213e,color:#e0e0e0
    style Providers fill:#0f3460,color:#e0e0e0

Features

Core Design

• Flow-file driven: Every production pipeline is defined as an SOP Markdown file. Change the SOP to change the output — no code changes required.

• No hardcoded brain: ShotFlow provides tools, not decisions. External agents read the SOP and orchestrate independently.

• SIMULATE mode: Develop and test the full pipeline without a GPU or API credentials. All providers return placeholder assets.

Cinematic Prompt System

• 13 style presets: cinematic, cyberpunk, anime, ink_wash, ghibli, oil_painting, realistic, watercolor, documentary, wes_anderson, scifi, fantasy, noir — each injects professional image/video suffixes and negative prompts into the LLM system prompt.

• 10 scene templates: product, food, travel, knowledge, story, city, nature, action, interview, tutorial — each defines shot rhythm, shot sequence, lighting, and transition style.

• Cinematic keyword library: 15 lighting types, 15 camera angles, 15 camera movements, and 15 mood keywords — sampled to enrich fallback prompts when no LLM is configured.

• Quality levels: standard (1080p), hd (1080p + bokeh), 4k (4K HDR ACES), 8k (8K HDR Dolby Vision) — controls technical parameters embedded in prompts.

Advanced Video Pipeline (FFmpeg)

• xfade transitions: 13 effects (fade, wipeleft, circleopen, distance, zoomin, smoothup, etc.) for cross-dissolves between segments.

• Ken Burns effect: a zoompan filter for static images — slow zoom in/out with alternating directions for visual variety.

• Color grading: 5 presets (vintage, cross_process, teal_orange, high_contrast, warm_film) via FFmpeg curves + eq filters.

• 60s+ long video: an xfade chain with offset calculation supports an unlimited segment count for coherent long-form output.

Open-Source AI Alternatives

Every open-source tool degrades gracefully — if it is not installed, the pipeline logs a warning and continues without crashing.

Provider Support

• 13 providers integrated behind a uniform BaseProvider ABC (12 cloud + 1 open-source).

• MCP + REST dual exposure: Both protocols are available for broad agent-framework compatibility.

• Easy to extend: Add a provider by implementing generate(kind, params) and registering it in app/services/providers/__init__.py.

Reproducibility

• Every generation step saves a complete Spec record to the database, capturing parameters, provider, and output asset references.

• Results can be re-examined, compared, and re-run.

• The project ships a changelog and full version control.

Agent Ecosystem Ready

• WorkBuddy skill: shotflow-driver generates a video from a single sentence.

• MCP manifest: Drop integration/shotflow.mcp.json into any MCP client to discover all 6 tools.

• OpenAPI spec: Import integration/openapi.json into code generators (OpenAPI Generator, Postman, etc.).

Supported Providers

All providers support SIMULATE_MODE=true — set this in .env to exercise the full pipeline without any keys.

Quick Start

Option A: Docker (Recommended)

git clone https://github.com/weed33834/ShotFlow.git
cd ShotFlow
docker compose up -d

• Frontend: http://localhost:3000

• Backend API: http://localhost:8000

• API Docs: http://localhost:8000/docs

SIMULATE_MODE is enabled by default — no API keys required.

Option B: Local Development

Prerequisites

• Python 3.10+

• Node.js 22+ (for frontend development)

• FFmpeg (for video assembly)

• (Optional) PostgreSQL for production

1. Clone and Set Up

git clone https://github.com/weed33834/ShotFlow.git
cd ShotFlow

# Backend
python -m venv venv
source venv/bin/activate  # Linux/macOS
# venv\Scripts\activate   # Windows
pip install -r backend/requirements.txt

# Environment
cp .env.example .env
# Edit .env if you have API keys; SIMULATE_MODE=true works out of the box

2. Initialize Database

PYTHONPATH=backend python backend/init_db.py

3. Start the Server

# Backend API
PYTHONPATH=backend uvicorn app.main:app --reload --port 8000

# Frontend (separate terminal)
cd frontend
npm install
npm run dev

4. Generate a Video (SIMULATE)

curl -X POST http://localhost:8000/api/v1/generate \
  -H "Content-Type: application/json" \
  -d '{
    "nl_prompt": "A happy little egg-yolk creature laughing on grass",
    "output_type": "video"
  }'

This runs the make_video.sop.md workflow in SIMULATE mode and returns a spec ID with placeholder asset URLs.

5. Verify MCP Server

PYTHONPATH=backend python -m app.services.mcp_server

The server logs FastMCP 3.4.4 and registers 6 tools, then waits for stdio-based agent communication.

Production Workflows

Each workflow is defined as an SOP Markdown file in flows/. The available flows and their step sequences are listed below.

Video Production (flows/make_video.sop.md)

flowchart LR
    A["1. Brainstorm<br/>Analyze prompt → subject detection"] --> B["2. Character Anchor<br/>Generate consistency anchor image"]
    B --> C["3. Multi-Shot Loop<br/>×3 shots"]
    C --> D["3a. Shot Image<br/>Hunyuan Image / Wanx"]
    C --> E["3b. Shot Video<br/>Kling / Hunyuan Video / Runway"]
    C --> F["3c. Shot Audio<br/>Tencent TTS"]
    D --> G["4. Lip Sync<br/>HeyGen"]
    F --> G
    G --> H["5. Assemble<br/>ffmpeg (placeholder in SIMULATE)"]

Image Set (flows/make_image_set.sop.md)

flowchart LR
    A["1. Brainstorm"] --> B["2. Character Anchor"]
    B --> C["3. Multi-Image Loop<br/>×3 frames"]
    C --> D["Image Generation<br/>Hunyuan Image / Wanx / Jimeng / NovelAI"]
    D --> E["4. Side-by-side Assembly"]

Comic / Dynamic Comic (flows/make_comic.sop.md)

flowchart LR
    A["1. Brainstorm"] --> B["2. Character Anchor"]
    B --> C["3. Panel-by-Panel<br/>×3 panels"]
    C --> D["Panel Image<br/>Hunyuan Image"]
    C --> E["Panel Audio<br/>Tencent TTS"]
    D --> F["4. Comic Assembly<br/>ffmpeg"]

Micro-Movie (flows/make_micro_movie.sop.md)

flowchart LR
    A["1. Brainstorm"] --> B["2. Character Anchor"]
    B --> C["3. Multi-Shot<br/>×3 shots"]
    C --> D["Image → Hunyuan Image"]
    C --> E["Video → Hunyuan Video"]
    C --> F["Audio → Tencent TTS"]
    D --> G["4. Lip Sync → HeyGen"]
    F --> G
    G --> H["5. Assemble"]

Visual Novel (flows/make_vn.sop.md)

flowchart LR
    A["1. Brainstorm"] --> B["2. Character Anchor"]
    B --> C["3. Scene Loop<br/>×3 scenes"]
    C --> D["Scene Image<br/>Hunyuan Image"]
    C --> E["Scene Dialogue<br/>Tencent TTS"]
    D --> F["4. VN Assembly<br/>ffmpeg"]

MCP Tool Reference

ShotFlow exposes 6 tools through its MCP server (app.services.mcp_server).

MCP Transport

The server listens on stdio by default (standard MCP transport). To use a streamable HTTP transport, configure your MCP client to proxy through the ShotFlow REST API or use an SSE bridge.

MCP Manifest

Use integration/shotflow.mcp.json for zero-configuration discovery:

{
  "mcpServers": {
    "ShotFlow": {
      "command": "python",
      "args": ["-m", "app.services.mcp_server"],
      "env": {
        "PYTHONPATH": "backend",
        "SIMULATE_MODE": "true"
      }
    }
  }
}

Agent Integration

ShotFlow is designed to be driven by external AI agents. Three integration paths are available.

Path 1: WorkBuddy (via shotflow-driver skill)

The shotflow-driver skill is installed at ~/.workbuddy/skills/shotflow-driver/. When you tell WorkBuddy:

"用 ShotFlow 出一份奶龙视频"

It reads flows/make_nailong_video.sop.md, calls the 6 MCP tools in sequence, and returns the final assembled output.

Path 2: Any MCP Client (Tencent Yuanqi, Dify, etc.)

1. Copy integration/shotflow.mcp.json into your MCP client configuration.

2. The client auto-discovers all 6 tools.

3. The client reads the SOP flow files and orchestrates tool calls.

Path 3: REST API (Alibaba Bailian, custom agents)

• Full OpenAPI 3.0 spec: integration/openapi.json

• Base URL: http://localhost:8000/api/v1

• Key endpoints: /generate, /anchor, /assemble, /spec, /tools/assets

Edge Deployment

For latency-sensitive scenarios (preview rendering, real-time dialogue), consider deploying the MCP server to edge functions:

• Tencent EdgeOne Makers: Agent-native hosting with global CDN acceleration.

• Alibaba Function Compute: Deploy ShotFlow tools as stateless functions behind the MCP protocol, with confidential computing (TDX) for credential protection.

Project Structure

shotflow/
├── backend/
│   ├── app/
│   │   ├── api/v1/           # REST endpoints
│   │   ├── core/             # Config, security
│   │   ├── models/           # SQLAlchemy models
│   │   ├── prompts/          # Cinematic style/scene/keyword libraries
│   │   ├── schemas/          # Pydantic schemas
│   │   └── services/
│   │       ├── providers/    # 13 provider integrations
│   │       ├── mcp_server.py # MCP tool definitions
│   │       ├── orchestrator.py
│   │       └── tools_service.py
│   ├── tests/
│   └── requirements.txt
├── frontend/
│   └── src/
│       ├── api/              # API client
│       ├── layouts/          # App layout
│       ├── pages/            # Generate, Workflows, Assets
│       └── types/            # TypeScript types
├── flows/                    # SOP flow files
│   ├── make_video.sop.md
│   ├── make_image_set.sop.md
│   ├── make_comic.sop.md
│   ├── make_micro_movie.sop.md
│   ├── make_vn.sop.md
│   └── make_nailong_video.sop.md
├── integration/              # Exposure package
│   ├── shotflow.mcp.json
│   ├── openapi.json
│   ├── server_card.json
│   └── AGENT_INTEGRATION_GUIDE.md
├── .env.example
├── LICENSE
├── README.md
└── CHANGELOG.md

FAQ

Q: Does ShotFlow require a GPU? A: No. All generation is offloaded to cloud vendor APIs. For development and testing, SIMULATE mode returns placeholder assets without a GPU or keys.

Q: Can I add my own provider? A: Yes. Create a class inheriting from BaseProvider, implement generate(kind, params) returning AssetResult, and register it in app/services/providers/__init__.py.

Q: Is there authentication for the REST API? A: Not built-in. Use a reverse proxy (Nginx, Caddy) with authentication for production deployments.

Q: Does ShotFlow store generated content? A: Asset references (URLs, metadata) are stored in the database. The actual media files live on the vendor's platform or your configured storage.

Contributing

Contributions are welcome. Please read CONTRIBUTING.md and the Code of Conduct before submitting a pull request.

License

ShotFlow is open source under the MIT License. See LICENSE for the full text.

ShotFlow — SOP-driven AIGC, agent-native by design.