ffmpeg-agent

English | 简体中文

Process media from applications or remote AI agents through HTTP and MCP. Upload a file, submit an asynchronous task, and download the result. Shared volumes are optional for clients; you can combine them with HTTP transfers.

Already have a service? Install the consumer skills. Need to run your own service? Build and start it with Docker.

Tasks can chain operations, reuse inputs and return multiple outputs. Start with the defaults by omitting params, then add only the settings you need. HTTP and MCP accept the same parameters. Query GET /v1/capabilities for available encoders, parameter schemas and limits.

Optional natural-language tasks turn a description into a validated media plan using PydanticAI. Enable the official A2A endpoint to expose the same task service to other agents. Both features are off by default. See the natural-language and A2A guide / 自然语言与 A2A 使用指南.

Connect your Agent to an existing service

Consumers need Python 3.10+, a reachable service URL and its service Token. No extra Python packages, Docker, FFmpeg, server source or model key are needed at runtime. Both skills include the same standalone standard-library client and can be installed independently.

Install and connect: copy this prompt to your Agent, replacing the URL with one reachable from its execution environment:

Install ffmpeg-agent-connect and ffmpeg-agent-media from https://github.com/breakstring/ffmpeg-agent/tree/main/skills.
Use this host's skill installation mechanism and install both complete directories, including scripts, references and agents metadata.
Then use ffmpeg-agent-connect to connect to https://media.example.com. Reuse my private consumer configuration; if it is missing, guide me to configure the service URL and Token in a private local file, without pasting the Token into chat.
Check the available protocols and media capabilities. Prefer the host's existing native remote MCP connection, and transfer local files over HTTP. Verify upload, media processing and a checked download using a small file I select.

For manual installation, use the complete commands and host directory table. Install the whole skill directory, not just SKILL.md, then refresh skills or start a new session as your host requires.

Start processing: after connecting, give your Agent the file and desired result:

Use ffmpeg-agent-media to convert this video to an H.264 MP4 no wider than 1280 pixels, preserving the source timing. Save it to a new output directory, verify the download, and retain the task state so I can resume if the wait times out.

Structured HTTP/MCP operations do not need a server model; your Agent can translate the request into supported parameters. Server-side natural-language planning and A2A are optional. Native MCP tools submit and query tasks; local file bytes still travel over HTTP. Installing a skill does not by itself register a native MCP connection. See the consumer guide for private configuration, native MCP handoff and runnable examples.

Related MCP server: Botverse

Start with Docker Compose

You need Git, Docker with Compose v2 and Python 3.10+ on the host for configuration and client scripts. Python 3.14, FFmpeg and media dependencies run inside the image; they do not need to be installed on the host.

git clone https://github.com/breakstring/ffmpeg-agent.git
cd ffmpeg-agent

Run all deployment commands below from this repository root.

First-time configuration

Initialize the configuration before starting the service for the first time:

python3 deploy/generate-config.py

The generator only prepares .env and deploy/local/media-config.json; it does not start the service. It fills missing settings, preserves your existing valid Token and configuration, and migrates legacy JSON credentials and feature switches into .env. The random service Token is stored only in .env, with mode 0600. Keep deploy/local private (0700); the examples read .env automatically, so Tokens do not need to appear in commands.

Build the image locally

Review .env and adjust any settings you need (see below), then build the image and start the service:

docker compose up -d --build --wait

Compose builds the local image ffmpeg-agent:v1-local, starts one service instance and waits for it to become ready. To run the build separately:

docker build -t ffmpeg-agent:v1-local .
docker compose up -d --no-build --wait

These commands use the default MEDIA_IMAGE in .env. If you changed it, use the same image name for the build and startup. The container runs as a non-root user with a read-only root filesystem; Compose provides persistent state, work and output volumes. Do not scale the service or share its SQLite state between instances.

Check the container and authenticated service connection, then try the upload → process → download example:

docker compose ps
python3 skills/ffmpeg-agent-connect/scripts/media_client.py --env-file .env check --protocol all

By default, the service is available at http://localhost:8080 on this host. Disabled A2A is expected until you enable it. A connection check does not replace actually processing a file.

Use a prebuilt GHCR image

Prebuilt Linux AMD64 and ARM64 images are available at ghcr.io/breakstring/ffmpeg-agent, with anonymous pulls enabled. Both architectures pass the full test suite and real HTTP/MCP media checks before publication. See the first release verification and publishing details.

This is an alternative to the local build above. Run the first-time configuration step, then set this non-secret value in .env:

MEDIA_IMAGE=ghcr.io/breakstring/ffmpeg-agent:latest

Pull and start the image without building locally:

docker compose pull media
docker compose up -d --no-build --wait media

Use the same two commands to update a registry deployment after draining active tasks and backing up its state. Keep --no-build when using a GHCR image; --build is for your local source image.

The release workflow validates both architectures before publishing the tested images. Tags in this table describe the publication scheme, not a promise that a particular release already exists. Choose an actual published tag or digest from the package page.

Routine startup

After initialization and a successful build or pull, start with the existing configuration:

docker compose up -d --no-build --wait

You do not need to run generate-config.py again for routine startup. For a local source image, add --build when source code, dependencies or the Dockerfile change. For a registry image, use the pull/start commands above.

Configuration and legacy migration

Configure the service Bearer Token, model key, model settings and feature switches in .env. Docker Compose explicitly forwards these settings to the service. The structural JSON contains storage, source rules and service limits; it must not contain service_token, model credentials or feature switches. If an existing installation still stores those settings in JSON, run the generator once before upgrading to migrate them. A process environment value overrides the corresponding .env setting; an empty Token is rejected. Keep credentials out of Git, shell commands and logs. Do not run source .env, print resolved Compose configuration, or share container environment inspection output.

A Token must have at least 24 printable ASCII characters without spaces. Missing or empty Tokens stop startup; only MEDIA_SERVICE_TOKEN=null selects open access. To enable natural-language tasks or A2A, follow the enablement guide. After editing .env, recreate the container so the new values take effect:

docker compose up -d --force-recreate --wait media

If you change the port, update MEDIA_PUBLIC_BASE_URL in .env too. worker_concurrency in the structural JSON sets logical task concurrency (1–4, default 1). If you use a custom JSON path, select it with MEDIA_CONFIG_PATH; examples can read its address with --config when no environment URL or --base-url is provided.

The default service is reachable only on the host. For remote use, provide an HTTPS entry point, set MEDIA_PUBLIC_BASE_URL to the externally reachable URL, and forward the correct public Host header. MCP validates Host and Origin against that address. A remote example client needs only a private .env with MEDIA_PUBLIC_BASE_URL and MEDIA_SERVICE_TOKEN, selected with --env-file; it does not need the server's storage or deployment JSON. Process environment settings override this file, and --base-url overrides the connection address. Tokens are never read from JSON by the example clients.

Upload, process and download

The repository includes complete clients that upload a local file, wait for processing, and download and verify the result. All commands below use existing sample files and a new temporary output directory, so they can be copied and run again. Keep the source unchanged while uploading.

HTTP: upload and submit together

This Python 3 standard-library client needs no extra packages. By default, it converts video to H.264/MP4, preserving source dimensions and frame timing:

output_dir=$(mktemp -d)
printf 'Output directory: %s\n' "$output_dir"
python3 examples/http-workflow.py \
  tests/fixtures/video_h264_640x360.mkv "$output_dir/result.mp4"

It sends one multipart request to POST /v1/jobs/upload, polls the returned task, and saves the verified MP4. You do not need to supply encoding parameters, handle IDs manually, or mount a shared volume.

HTTP: upload first, then submit

Add --upload-first to use POST /v1/files followed by POST /v1/jobs. The client also prints the file_id, which can be reused in other tasks while the upload is available:

output_dir=$(mktemp -d)
printf 'Output directory: %s\n' "$output_dir"
python3 examples/http-workflow.py \
  tests/fixtures/video_h264_640x360.mkv "$output_dir/result.mp4" \
  --upload-first

Both modes complete the same upload → process → download workflow. When building your own HTTP client, the multipart job part has this shape (a request-body reference, not a standalone command):

{
  "inputs": {"source": {"kind": "upload", "part": "source"}},
  "steps": [{
    "id": "process",
    "operation": "video.transcode.v1",
    "inputs": {"video": {"source": "source"}}
  }],
  "outputs": {"result": {"step": "process", "output": "video"}}
}

Send job first, then the source file part, with an Idempotency-Key header. For upload-first submission, replace the input with {"kind":"uploaded_file","file_id":"<returned file_id>"}. See the HTTP contract for the endpoints and client source for a complete implementation.

Use remote MCP

With uv installed, this command runs the complete MCP client with the SDK version pinned by the project:

output_dir=$(mktemp -d)
printf 'Output directory: %s\n' "$output_dir"
uv run --frozen python examples/mcp-workflow.py \
  tests/fixtures/video_h264_640x360.mkv "$output_dir/result.mp4"

The client uploads over HTTP, calls the remote transcode_video tool, waits through MCP, and downloads over HTTP. It reads the same .env Token for both transports. There is no local stdio server to configure.

To connect another MCP client, use the Streamable HTTP endpoint at MEDIA_PUBLIC_BASE_URL + /mcp (locally http://localhost:8080/mcp) and configure the service Bearer header in its connection settings. File HTTP requests need the same header. MCP alone cannot upload a path from your computer: the client needs HTTP file-transfer capability, or an input URL/shared file the service can access.

For reference, these are the arguments to transcode_video after HTTP upload; replace the placeholders with the returned file_id and a key for this request:

{
  "source": {"kind": "uploaded_file", "file_id": "<returned file_id>"},
  "idempotency_key": "<key for this request>"
}

The tool returns job_id. Call get_job until it finishes, then use outputs.result.artifact_id with get_artifact to obtain download information. The runnable client handles these steps for you. Discover all tools through tools/list; the six media tools share their parameter models with HTTP, and the remaining tools manage tasks, files, artifacts and leases. See the MCP contract.

Try the six operations

Both clients accept the same --operation and --params options. The default operation is transcode_video; all six work without --params.

For example, download a frame collection with one command:

output_dir=$(mktemp -d)
printf 'Output directory: %s\n' "$output_dir"
python3 examples/http-workflow.py \
  tests/fixtures/video_h264_640x360.mkv "$output_dir/frames" \
  --operation extract_frames

Use uv run --frozen python examples/mcp-workflow.py instead of python3 examples/http-workflow.py to run the same operation through MCP.

Adjust only what you need

A parameters file contains just the values to override. This complete example fits the video within 1920×1080 without enlarging it, uses CRF 20, and encodes mono AAC at 44.1 kHz:

output_dir=$(mktemp -d)
printf 'Output directory: %s\n' "$output_dir"
cat > "$output_dir/params.json" <<'JSON'
{
  "resize": {"width": 1920, "height": 1080},
  "rate_control": {"mode": "crf", "value": 20},
  "audio": {"sample_rate": 44100, "channels": 1}
}
JSON
python3 examples/http-workflow.py \
  tests/fixtures/video_h264_640x360.mkv "$output_dir/result.mp4" \
  --params "$output_dir/params.json"

The same file works with the MCP client. In your own requests, put this object in the HTTP step's params or the MCP tool's params.

Available codecs, formats and common values

These values are supported by the service's parameter contract; use lowercase strings. codec selects the encoding, container selects the file container, and images use format.

Common combinations are H.264 + MP4, HEVC + MP4, VP9/AV1 + WebM, AAC + M4A, MP3 + MP3, Opus + Ogg, and PCM + WAV. Changing only codec selects a compatible default container. Explicit combinations must match; H.264 + WebM, for example, is rejected.

Actual availability is the intersection of service support and runtime support, subject to combination rules and deployment limits. Docker uses FFmpeg/Pillow inside the container; installing codecs on the host does not add service capabilities. The current implementation uses software encoding. Query the capabilities endpoint below for your deployment. See the media parameter reference for complete compatibility tables, enum meanings and 10 ready-to-use JSON parameter examples.

Presets are optional shortcuts: {"preset":"web_720p_v1"} selects 720p/30 fps with stereo AAC; aac_stereo_128k_v1 selects AAC 128 kbps/48 kHz/stereo; wav_mono_16k_v1 selects 16 kHz mono PCM. Explicit settings override the selected preset. Without a preset, video preserves source dimensions and timing. Audio auto prefers source sample rate/channels and falls back to compatible values; source requires exact preservation. Audio copy mode preserves the original encoding and cannot be combined with encoding settings or trimming.

Query GET /v1/capabilities or MCP list_capabilities for available encoders, schemas, defaults and limits. Accepted settings appear in task steps[].effective_params; resolved output settings appear in artifact media.effective_params. Full codec, size, frame-rate, bitrate, sample-rate and quality controls are documented in the parameter contract.

Waiting, retries and downloads

The clients print a request ID and then the accepted job_id. Each invocation creates a new request by default. Reuse an explicit --request-id only to retry the same complete request, including source, operation, parameters and upload mode. Use a new ID when changing any of them.

The default wait is 600 seconds; change it with --wait-seconds. A wait timeout leaves processing running. To resume, run the same client with --job-id and the printed ID; it skips upload and submission. SOURCE DESTINATION are still required, but the source is not opened during resume. Use --help for all options. --env-file selects the private environment file (default: repository .env); --base-url overrides the connection URL. --config only supplies a fallback public_base_url, never a Token.

A destination file or frame directory must not already exist. Downloads are streamed and checked against the advertised size and SHA-256 before a completed file appears. Frame collections include manifest.json with successful and failed members. A partial result saves available members and exits with status 2; a failed task or transfer exits nonzero. The clients do not overwrite existing files or treat an incomplete transfer as a complete result.

Choose an input and consume results

HTTP sources are denied by default until configured. Shared input files are borrowed read-only and never deleted by the service. Keep them unchanged until every consuming task reports inputs_released=true. Shared volumes do not disable uploads, HTTP downloads or MCP.

Tasks finish as succeeded, partial, failed or canceled; inspect each declared output before using it. POST /v1/jobs/{job_id}/cancel requests cancellation. Wait for the terminal state and inputs_released=true before changing or deleting shared inputs.

Single-file artifacts expose GET/HEAD /v1/artifacts/{artifact_id}/content, including Range and ETag support. Frame collections expose a paginated member list and individual file downloads; collections are not automatically zipped.

Use POST /v1/files/{file_id}/release or POST /v1/artifacts/{artifact_id}/release for early reclamation. Release stops new references and waits for existing users. For shared-output reads, obtain a lease before opening the file and release it afterward. HTTP transfers protect their active sends; an explicit lease also protects the interval between metadata retrieval, download and later consumption. Retention and lease periods are configurable; exact expiration times are returned by the API.

Operate and configure

docker compose stop preserves volumes. Before backups or upgrades, stop new submissions, let tasks drain, then stop the service. Back up state, work, output, private .env, structural JSON and image version together; your application owns shared-input backups. Avoid docker compose down -v for routine stops or cleanup.

Run one service instance with its own state and work volumes. The default is one logical Worker, configurable from 1 to 4; increasing it requires workload and resource testing. Do not run several containers against the same state directory.

The current processing scope excludes HDR, 10-bit video, hardware acceleration, non-square SAR, animated/multipage images, HLS processing and arbitrary FFmpeg commands. See the API contract for supported media semantics and errors.

• Deployment and shared-volume examples

• HTTP API contract and OpenAPI

• Implementation and verification status