Mage-VL GGUF conversion and local inference

Local MCP video understanding

This repository now packages the patched Mage-VL GGUF runtime as a local MCP service. An MCP-capable agent can queue a video task, read durable events with a cursor, and keep reasoning over the returned evidence without ever uploading the video to a cloud service.

The supported topology is deliberately split in two:

Agent / MCP client ──HTTP MCP──> WSL MCP orchestrator (127.0.0.1:8765/mcp)
                                      │ SQLite WAL + one FIFO worker
                                      ▼
                           Docker CUDA Mage runtime (127.0.0.1:8080)
                                      │
                                local video files

The runtime is the existing patched GGUF implementation; this project does not re-create or quantize Mage-VL weights. The default profile is designed for an 8 GiB NVIDIA GPU: Q4_K_M language backbone, Q8 vision/StreamMind sidecars, F16 DCVC sidecars, an 8192-token context, and Q4 KV cache.

Prerequisites

• Windows with WSL2 and an NVIDIA driver visible in nvidia-smi inside WSL.

• Ubuntu 24.04 WSL distribution named Ubuntu-24.04 (or pass -Distro).

• At least 20 GiB free in Docker's WSL filesystem before the first image build.

• Persistent data directory E:\mageVL-data; model weights, cache, SQLite database, logs, and the generated runtime configuration remain there.

The host used during development has an RTX 4060 Laptop GPU with 8188 MiB; do not assume a different machine has the same headroom. The 12.8.1 CUDA image is the default. runtime.env exposes CUDA_VERSION=12.6.3 as a manual fallback if a compatible Docker runtime cannot start it; the scripts never update a Windows graphics driver.

Install and run

Open PowerShell 7 in this repository and run:

.\scripts\mage-vl-mcp.ps1 setup-system
# Close/reopen the WSL shell after Docker group membership is applied.
.\scripts\mage-vl-mcp.ps1 setup-runtime
.\scripts\mage-vl-mcp.ps1 start

setup-system is intentionally interactive: it installs Docker Engine and NVIDIA Container Toolkit inside WSL and may request the Linux sudo password. setup-runtime downloads six pinned GGUF artifacts from JohnTdi/Mage-VL-GGUF revision 63b23eb4707b1907668c57d61845e7d423016b5c, writes E:\mageVL-data\models\SHA256SUMS.txt, installs the MCP package in a repository-local virtual environment, and builds the CUDA image.

start stays in the foreground. Pressing Ctrl+C stops the MCP supervisor but keeps the warm Docker runtime running. Use these additional commands:

.\scripts\mage-vl-mcp.ps1 status
.\scripts\mage-vl-mcp.ps1 stop
.\scripts\mage-vl-mcp.ps1 stop -All

stop -All stops both the supervisor and the runtime container; it retains all data below E:\mageVL-data.

Local-file boundary

The MCP service does not expose URLs, RTSP, cameras, screens, or arbitrary container paths. Before starting it, edit the generated E:\mageVL-data\mcp.env and set MAGE_VIDEO_ROOTS to one or more existing WSL directories, separated by commas. The default is /mnt/e/mageVL-data/videos.

Every submitted video_path is converted from a Windows drive path when needed, resolved through symlinks, and rejected unless it remains below an allowed root. For example, add /mnt/e/Videos only if that is the directory you intend agents to inspect.

MCP client endpoint and tools

Use this Streamable HTTP endpoint in a local MCP client:

http://127.0.0.1:8765/mcp

All six inference tools share one FIFO channel. A watch session blocks offline analysis and segment inspection until it finishes or is stopped. This is intentional: the native StreamMind runner swaps out ordinary llama-server and holds recurrent state. On an orchestrator restart, in-progress jobs are marked failed rather than silently resumed.

Offline analysis asks Mage for structured JSON. If model output is not valid JSON, the raw response is preserved in the event instead of being discarded or presented as a fabricated timeline.

What CI and tests prove

tests/test_mcp_orchestrator.py exercises the path boundary, SQLite event cursor, and queued cancellation behavior. GitHub Actions additionally checks the pinned native patch applies and compiles the CPU llama-mage-codec-stream target. These checks do not prove a real CUDA container, a model download, or end-to-end video inference; run setup-runtime and a local video task for that validation.

See CONTEXT.md for the domain vocabulary and docs/adr for the two architectural decisions.

Related MCP server: popcorn

Native StreamMind gate

This fork executes Microsoft's proactive StreamMind path entirely inside llama.cpp: Mage-ViT embeddings are grouped by codec timestamp, averaged over patches, passed through the stateful Mamba-1 EPFE, and scored by the four-layer Qwen3 gate classifier. It does not reconstruct Transformers tensors or keep a second BF16 model in VRAM.

cmake -S llama.cpp -B llama.cpp/build -DGGML_VULKAN=ON -DLLAMA_BUILD_EXAMPLES=ON
cmake --build llama.cpp/build --target llama-streammind-e2e -j
GGML_VK_VISIBLE_DEVICES=0 llama.cpp/build/bin/llama-streammind-e2e \
  models/mage-vl-backbone-Q8_0.gguf models/mage-vit-mmproj-Q8_0.gguf \
  models/mage-streammind-epfe-Q8_0.gguf models/mage-streammind-cls-Q8_0.gguf \
  video.mcv

Each JSONL row contains the source frame, official silent/speak logits, speak probability, and the raw decision at Microsoft's 0.5 boundary. The gate is application-neutral: clients decide what a speak event means and may apply their own policy. STREAMMIND_CHUNK=N processes input incrementally while preserving recurrent state.

MP4, RTSP and HLS input

streammind_native.py is a transport/preprocessing adapter. In incremental mode one persistent FFmpeg process decodes the stream and the open readiness selector builds Mage canvases as evidence becomes sufficient; neither executes a neural model. Mage-ViT, the Mamba-1 EPFE and gate classifier all execute in the patched C++ llama.cpp runtime, so no Transformers checkpoint or BF16 duplicate is loaded.

Install only the video preprocessor environment, then keep it active so codec-video-prep and cv-preinfer are on PATH:

python3.12 -m venv .venv-codec
source .venv-codec/bin/activate
pip install "codec-video-prep>=0.2.5"

Local MP4:

python tools/streammind_native.py video.mp4 \
  --runner llama.cpp/build/bin/llama-streammind-e2e \
  --backbone models/mage-vl-backbone-Q8_0.gguf \
  --mmproj models/mage-vit-mmproj-Q8_0.gguf \
  --epfe models/mage-streammind-epfe-Q8_0.gguf \
  --classifier models/mage-streammind-cls-Q8_0.gguf \
  --incremental-producer tools/live_codec_stream.py \
  --vulkan-device 0

RTSP camera and HLS use the same command; only the source changes:

python tools/streammind_native.py 'rtsp://user:password@camera/stream1' ...
python tools/streammind_native.py 'https://host/live/playlist.m3u8' ...

Incremental live mode has no fixed transport segment and writes no temporary MP4. At the default 8 sampled FPS it can close after the minimum eight samples (one second) when readiness and temporal coverage are satisfied; otherwise it extends up to --sampled-frames. EPFE state continues across every adaptive group until the process exits. For local files, --realtime makes FFmpeg feed frames at playback speed. Small MAGECV1 handoff bundles are deleted immediately after consumption. Omitting --incremental-producer retains the segmented codec-bitcost compatibility path for offline/reference work.

Repeatable conversion files and local-inference instructions for microsoft/Mage-VL. The released GGUF weights were measured on image, video and language benchmarks and compared with Microsoft's reported BF16 reference values.

Model weights: JohnTdi/Mage-VL-GGUF on Hugging Face

Mage-VL Studio: native Q8 GGUF analysis with a selected time range, dedicated static-text OCR, RAM/VRAM metrics and representative full-frame Highlights.

Development disclosure: code assistance and review were provided by OpenAI GPT-5.6 Sol. Final integration, testing and release decisions were made and verified by the repository maintainer.

This GitHub repository contains the Docker runtime, patches and launch instructions. The Hugging Face repository contains the Q4/Q8 backbone and F16/Q8 vision GGUF artifacts.

Status

The Docker images apply a small native runtime patch to pinned llama.cpp. Both the language backbone and Mage-ViT remain in their GGUF storage types during inference; no Transformers process or BF16 reconstruction is involved.

Released variants

Weights are intentionally not stored in Git. All eight runtime artifacts are in the JohnTdi/Mage-VL-GGUF model repository.

Quick start: native llama.cpp server

One-command guided installation

After cloning the repository, the installer detects CUDA or Vulkan, estimates VRAM, selects an 8/16/24–32 GB profile, downloads only the required GGUF files, derives the matching DRM nodes/groups and starts Docker:

./install.sh

Override detection with MAGE_BACKEND=vulkan|cuda and MAGE_PROFILE=8|16|24|32. The generated .env remains editable.

The included images compile a pinned llama.cpp revision, apply the Mage-ViT runtime patch, and include the image/video dependencies. The unused upstream llama.cpp Web UI is disabled at build time; this avoids Node/npm and mutable UI downloads, while the gateway provides its own local upload page.

This repository is the complete runtime distribution: Docker clones pinned llama.cpp, applies the unified native Mage patch from patches/, compiles it, and starts llama-server with both GGUF files. A separate llama.cpp fork checkout is not required.

Requirements: Linux, Git, Python 3 with venv and pip, Docker Engine, and Docker Compose 2.30 or newer. Start from an empty directory:

git clone https://github.com/JohnTDI-cpu/mage-vl-gguf.git
cd mage-vl-gguf

python3 -m venv .hf-venv
.hf-venv/bin/pip install "huggingface_hub>=0.34"
.hf-venv/bin/hf download JohnTdi/Mage-VL-GGUF \
  mage-vl-backbone-Q8_0.gguf mage-vit-mmproj-Q8_0.gguf \
  mage-streammind-epfe-Q8_0.gguf mage-streammind-cls-Q8_0.gguf \
  mage-dcvc-rt-intra-F16.gguf mage-dcvc-rt-inter-F16.gguf \
  --local-dir models

cp .env.example .env
# RADV needs both DRM nodes from the same GPU. Keep their host names unchanged.
sed -i "s/^RENDER_GID=.*/RENDER_GID=$(stat -c '%g' /dev/dri/renderD128)/" .env
sed -i "s/^VIDEO_GID=.*/VIDEO_GID=$(stat -c '%g' /dev/dri/card0)/" .env
docker compose --profile vulkan up -d --build --wait vulkan
curl --fail http://localhost:8080/health

The first build compiles our patched llama.cpp and may take several minutes. When /health succeeds, open http://localhost:8080 in a browser, choose an MP4, enter a question and click Analyze video. No manual conversion or codec command is required.

For scripts, upload a JPEG/PNG image:

curl --fail http://localhost:8080/v1/image/analyze \
  -F image=@./your-image.jpg \
  -F 'prompt=Is there a person in this image? Answer yes or no.' \
  -F max_tokens=32

Or upload an ordinary MP4. H.264 and HEVC go directly to the official codec-aware preprocessor; AV1, VP9, MPEG-4 Part 2 and other FFmpeg-readable video codecs are converted to high-quality H.264 automatically. The container then packs MAGECV1 and runs native GGUF inference:

curl --fail http://localhost:8080/v1/video/analyze \
  -F video=@./your-video.mp4 \
  -F 'prompt=Describe the important events in temporal order.' \
  -F max_tokens=256

Continuous live monitoring

Open Mage-VL Studio, select Live stream, paste an RTSP/RTMP, direct HTTP/HLS, localhost or supported page URL such as YouTube, configure the response policy, and select Start live analysis. The native C++ process performs FFmpeg decode, temporal sampling, DCVC-RT GGUF and Mage canvas construction, then preserves StreamMind's recurrent EPFE state across groups. Mage-ViT encodes a group once; the same embeddings feed the gate and any triggered Qwen response. Results with timestamps appear immediately below the player.

Demonstration using a randomly selected public YouTube live stream; the source was chosen only to exercise the live-analysis path and is not an endorsement.

The continuous path creates no transport MP4 or MAGECV1 handoff files. Stop the live session before changing model settings. Source and processing latency depend on network and frame content. The UI reports live delay, p95 real-time factor (RTF), pending/dropped windows and whether the stream is keeping pace. Drop stale windows is enabled by default so an overloaded installation stays current instead of analyzing an ever-growing history.

Live FPS is currently explicit, not automatically benchmarked or adapted to the user's GPU. The Analyzed FPS field defaults to 8; the hardware profile sets a conservative starting point, but it does not change FPS while a session is running. If the pipeline falls behind, the bounded queue drops stale windows when Drop stale windows is enabled instead of accumulating unlimited delay. Use the reported RTF and queue telemetry to tune it: keep p95 RTF below 0.8, reduce analyzed FPS (12 -> 8 -> 6 -> 4 -> 2) first, and then lower MAGE_DCVC_LIVE_MAX_HEIGHT (720 -> 480 -> 360) if necessary. Increase either setting only after the stream remains stable for several minutes.

Measured R9700 capacity results are in docs/live-performance-r9700.md. On this GPU, the safe default accepts a 1080p/4K source but downsizes it to 480p and samples 8 fps. Native DCVC measured 14.30 fps at 854x480, 6.34 fps at 720p and 1.55 fps at 1080p. The first empty-cache start can spend 15-17 seconds compiling Vulkan graphs; warmed sessions do not repeat that cost.

Stop the service with docker compose --profile vulkan down. Subsequent starts can omit --build:

docker compose --profile vulkan up -d --wait vulkan

Choose a quantization

Set the pair in .env; all four combinations are supported:

# Highest GGUF quality
GGUF_FILE=mage-vl-backbone-Q8_0.gguf
MMPROJ_FILE=mage-vit-mmproj-F16.gguf

# Recommended compact setup
# GGUF_FILE=mage-vl-backbone-Q4_K_M.gguf
# MMPROJ_FILE=mage-vit-mmproj-Q8_0.gguf

Download every variant if you want to switch without downloading later:

.hf-venv/bin/hf download JohnTdi/Mage-VL-GGUF \
  mage-vl-backbone-Q8_0.gguf mage-vl-backbone-Q4_K_M.gguf \
  mage-vit-mmproj-Q8_0.gguf mage-vit-mmproj-F16.gguf \
  mage-streammind-epfe-Q8_0.gguf mage-streammind-cls-Q8_0.gguf \
  mage-dcvc-rt-intra-F16.gguf mage-dcvc-rt-inter-F16.gguf \
  --local-dir models

The API binds to 127.0.0.1 by default because llama-server has no authentication in this configuration. To expose it deliberately, set HOST_BIND in .env and protect it with a firewall or authenticated reverse proxy. Never forward the raw port directly to the Internet.

The default configuration exposes only the matching /dev/dri/renderD128 + /dev/dri/card0 pair to the container, so another Vulkan GPU is not visible. Verify both against /dev/dri/by-path before first use. If you change them, also derive RENDER_GID and VIDEO_GID from the same nodes. Do not remap them to different names inside the container: RADV follows their sysfs relationship and can fail authentication when names are rewritten. An NVIDIA image is available with --profile cuda; it needs the NVIDIA Container Toolkit. See docker/README.md.

The public gateway exposes only the local upload page, /health, /v1/image/analyze, /v1/video/prepare, /v1/video/analyze-prepared, /v1/video/analyze, the /v1/live/sessions session API and read-only codec-canvas previews; the internal llama-server listens only inside the container. Preprocessed videos are cached by content hash and all output-affecting preprocessing settings, so a repeated upload skips both transcoding and codec preprocessing. Video audio tracks are deliberately ignored: Mage-VL analyzes visual content, not speech or sound.

The Live Stream panel accepts RTSP/RTMP, direct HTTP/HLS, localhost URLs and supported web pages such as YouTube (resolved inside the container with yt-dlp). It offers four response policies: periodic, every detected change, important changes only, and important changes plus a periodic report. Window length, minimum response gap, importance sensitivity, visual quality, response length, RTSP transport and the user prompt are configurable. Closed transport segments and their MAGECV1 workspaces are deleted immediately after inference; only the bounded in-memory result history remains. Stop a session explicitly before changing model settings.

The optimized native live runtime encodes every video window with Mage-ViT exactly once. Its embeddings feed both StreamMind EPFE/classification and, only after a trigger, Qwen generation. Starting live mode unloads the ordinary upload llama-server; stopping live restores it, so two language backbones are never resident simultaneously. Each live result exposes vision_encode_count=1 and shared_vision_embeddings=true as a runtime check. The Docker release includes the complete native StreamMind patch and both Q8 EPFE/classifier sidecars are downloaded from the linked Hugging Face repository.

The browser panel previews the local MP4 immediately, shows preprocessing and inference progress, renders the model answer and PP/TG metrics, and displays the exact codec canvases passed into Mage-ViT. A canvas is a spatial mosaic of selected source-frame patches, not necessarily a conventional full frame. Every card therefore reports its exact source-frame timestamp range and the complete timestamp list derived from src_patch_position.npy. The player includes a two-handle analysis-range selector. Only the selected interval is decoded and cached, while every preview timestamp is translated back to the absolute timeline of the original video. Its five-step Speed/Detail control changes both temporal sampling (96–320 frames) and the codec-canvas pixel budget (90k–180k). The optional thorough text scan deliberately runs a separate OCR-focused inference pass: experiments showed that a single general event prompt can omit a readable static caption even at the highest visual-detail setting.

Advanced settings can reload the model with a different context, batch, micro-batch and F16/Q8/Q4 KV cache after an explicit confirmation. If the new configuration cannot start, the gateway attempts to restore the previous one. The resource panel reports combined gateway+llama resident RAM on Linux. On NVIDIA it uses per-process nvidia-smi VRAM; on AMD/Vulkan, where the kernel does not expose reliable per-process VRAM, it reports the selected DRM device's increase over the pre-load baseline and labels that method explicitly.

Defaults for a 16 GiB GPU

The shipped profile uses one request slot, F16 KV cache and ctx=16384. On a Radeon AI PRO R9700 with Vulkan, the 50.64-second 1080x1920 H.264 test clip described in the benchmark repository produced 8,099 prompt tokens:

The runtime prefill patch combines successive timestamp-text and visual spans until the decoder batch is full. Before this patch, the same input created many small Vulkan submissions and reached only 1,221 tok/s. With the patch and ctx=32768 it reached 2,718 tok/s and used about 10.4 GiB; reducing the preallocated context to 16k retains the same F16 KV values while saving roughly 2.8 GiB. Exact values depend on driver, prompt, canvas count and power state.

Safety/resource defaults live in .env: 16,384 context tokens, one concurrent preprocessor, 2 GiB upload, 60-minute duration, 3840x2160 source video, 256 sampled frames, 150,000 pixels per video canvas, 256 generated tokens, and a 15-minute preprocessing timeout. JPEG/PNG images are automatically reduced to at most 1,048,576 pixels to keep image and video workloads within the 16 GiB profile. The content-addressed preprocessing cache is capped at 50 GiB and evicts least-recently-used entries. Change the corresponding MAGE_* values in .env, then recreate the service with docker compose --profile vulkan up -d --force-recreate vulkan.

The most useful tuning variables are LLAMA_ARG_CTX_SIZE (context/VRAM), MAGE_SAMPLED_FRAMES (temporal coverage), MAGE_MAX_PIXELS (tokens per codec canvas), MAGE_IMAGE_MAX_PIXELS, and MAGE_MAX_NEW_TOKENS. Decoder submission is explicitly pinned to MAGE_BATCH_SIZE=2048, MAGE_UBATCH_SIZE=512 and F16 KV cache. If a custom image setting creates a visual chunk larger than the decoder batch, the API returns a clear 422 error instead of accepting a truncated prompt; increase MAGE_BATCH_SIZE or reduce image pixels.

Video duration is not mapped one-to-one to context: the default samples at most 256 frames and readiness grouping produced 24--52 canvases (about 4,800--10,400 visual tokens) across tested 15--85 second clips. More sampled frames or larger canvases increase preprocessing time, context use and memory. If a request cannot fit, lower MAGE_SAMPLED_FRAMES or MAGE_MAX_PIXELS; increase LLAMA_ARG_CTX_SIZE only when sufficient VRAM remains.

Convert to GGUF

The patch targets llama.cpp commit a52077c4cabb4f3c0298329c9d2dd1324d5604cb. A different revision may require manual conflict resolution. Run this block from the root of the cloned mage-vl-gguf repository; it creates llama.cpp/ beneath it.

Use a separate converter venv. Its pinned requirements install CPU PyTorch and must not replace the ROCm/CUDA environment used for inference.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout a52077c4cabb4f3c0298329c9d2dd1324d5604cb
git apply ../patches/llama.cpp-mage-native-streammind.patch
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements-convert_hf_to_gguf.txt

python convert_hf_to_gguf.py ../models/Mage-VL \
  --outfile ../mage-vl-backbone-BF16.gguf --outtype bf16
python convert_hf_to_gguf.py ../models/Mage-VL \
  --mmproj --outfile ../mage-vit-mmproj-F16.gguf --outtype f16
python convert_hf_to_gguf.py ../models/Mage-VL \
  --mmproj --outfile ../mage-vit-mmproj-Q8_0.gguf --outtype q8_0

Build llama.cpp and quantize the language backbone:

cmake -B build -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j
build/bin/llama-quantize ../mage-vl-backbone-BF16.gguf \
  ../mage-vl-backbone-Q8_0.gguf Q8_0
build/bin/llama-quantize ../mage-vl-backbone-BF16.gguf \
  ../mage-vl-backbone-Q4_K_M.gguf Q4_K_M

llama.cpp backbone performance

Radeon AI PRO R9700, Vulkan, llama.cpp a52077c, batch 2048, ubatch 512, Flash Attention enabled:

These figures measure the Qwen3 language backbone, not codec preprocessing or Mage-ViT. Our released GGUF variants were measured on the complete MMBench EN dev set (4,329 records), Video-MME tc32 without subtitles (2,700 questions), WikiText-2 and a nine-image numerical vision comparison. Q8 + vision Q8 scored 84.36% MMBench CircularEval and 63.33% Video-MME. Microsoft's reported BF16 reference values are 84.19% and 64.00%, respectively. The Hugging Face model card contains the complete comparison, protocol summary and checksums.

Native validation

The current Vulkan Docker build passes 10 images plus 10 H.264 videos for each release combination: Q4+vision Q8, Q8+vision Q8, Q4+vision F16 and Q8+vision F16 — 80/80 deterministic semantic checks. The reusable harness is tests/native_sanity.sh. Gateway tests also cover image and cached-video inference for all four pairs, AV1-in-MP4 conversion, malformed MAGECV1 rejection, cache reuse, health propagation, graceful shutdown and native live stream processing. These execution checks complement MMBench and Video-MME. The release patch chain is compiled on every push by .github/workflows/ci.yml; GPU quality/performance checks remain release-gate tests because hosted CI has no suitable Vulkan/CUDA device or model weights.

License and upstream projects

Mage-VL is Apache-2.0 licensed. llama.cpp is MIT licensed. This repository contains integration patches and documentation; the upstream licenses continue to apply to their respective code and model artifacts. The NOTICE separates community code, upstream runtime and model terms.