# Mage-VL GGUF conversion and local inference

## Local MCP video understanding

This repository now packages the patched Mage-VL GGUF runtime as a **local MCP
service**. An MCP-capable agent can queue a video task, read durable events with
a cursor, and keep reasoning over the returned evidence without ever uploading
the video to a cloud service.

The supported topology is deliberately split in two:

```text
Agent / MCP client ──HTTP MCP──> WSL MCP orchestrator (127.0.0.1:8765/mcp)
                                      │ SQLite WAL + one FIFO worker
                                      ▼
                           Docker CUDA Mage runtime (127.0.0.1:8080)
                                      │
                                local video files
```

The runtime is the existing patched GGUF implementation; this project does not
re-create or quantize Mage-VL weights. The default profile is designed for an
8 GiB NVIDIA GPU: Q4_K_M language backbone, Q8 vision/StreamMind sidecars,
F16 DCVC sidecars, an 8192-token context, and Q4 KV cache.

### Prerequisites

- Windows with WSL2 and an NVIDIA driver visible in `nvidia-smi` inside WSL.
- Ubuntu 24.04 WSL distribution named `Ubuntu-24.04` (or pass `-Distro`).
- At least 20 GiB free in Docker's WSL filesystem before the first image build.
- Persistent data directory `E:\mageVL-data`; model weights, cache, SQLite
  database, logs, and the generated runtime configuration remain there.

The host used during development has an RTX 4060 Laptop GPU with 8188 MiB; do
not assume a different machine has the same headroom. The 12.8.1 CUDA image is
the default. `runtime.env` exposes `CUDA_VERSION=12.6.3` as a manual fallback
if a compatible Docker runtime cannot start it; the scripts never update a
Windows graphics driver.

### Install and run

Open PowerShell 7 in this repository and run:

```powershell
.\scripts\mage-vl-mcp.ps1 setup-system
# Close/reopen the WSL shell after Docker group membership is applied.
.\scripts\mage-vl-mcp.ps1 setup-runtime
.\scripts\mage-vl-mcp.ps1 start
```

`setup-system` is intentionally interactive: it installs Docker Engine and
NVIDIA Container Toolkit **inside WSL** and may request the Linux sudo
password. `setup-runtime` downloads six pinned GGUF artifacts from
`JohnTdi/Mage-VL-GGUF` revision `63b23eb4707b1907668c57d61845e7d423016b5c`,
writes `E:\mageVL-data\models\SHA256SUMS.txt`, installs the MCP package in a
repository-local virtual environment, and builds the CUDA image.

`start` stays in the foreground. Pressing Ctrl+C stops the MCP supervisor but
keeps the warm Docker runtime running. Use these additional commands:

```powershell
.\scripts\mage-vl-mcp.ps1 status
.\scripts\mage-vl-mcp.ps1 stop
.\scripts\mage-vl-mcp.ps1 stop -All
```

`stop -All` stops both the supervisor and the runtime container; it retains all
data below `E:\mageVL-data`.

### Local-file boundary

The MCP service does **not** expose URLs, RTSP, cameras, screens, or arbitrary
container paths. Before starting it, edit the generated
`E:\mageVL-data\mcp.env` and set `MAGE_VIDEO_ROOTS` to one or more existing
WSL directories, separated by commas. The default is
`/mnt/e/mageVL-data/videos`.

Every submitted `video_path` is converted from a Windows drive path when
needed, resolved through symlinks, and rejected unless it remains below an
allowed root. For example, add `/mnt/e/Videos` only if that is the directory
you intend agents to inspect.

### MCP client endpoint and tools

Use this Streamable HTTP endpoint in a local MCP client:

```text
http://127.0.0.1:8765/mcp
```

| Tool | Purpose |
|---|---|
| `analyze_video(video_path, language, force)` | Queue full-video analysis; compatible completed runs can be reused. |
| `start_video_watch(video_path, pacing, language)` | Run native StreamMind over a finite local video. `realtime` follows source PTS and never drops windows. |
| `get_job(job_id)` | Read queued/running/succeeded/failed/cancelled state. |
| `get_video_events(run_id, after_event_id, limit, wait_ms)` | Cursor-based event retrieval and optional local long-poll. |
| `inspect_video_segment(video_path, start_seconds, end_seconds, question, language)` | Queue a direct question over one bounded interval. |
| `stop_video_watch(run_id)` | Cancel a queued or running local-file watch session. |

All six inference tools share one FIFO channel. A watch session blocks offline
analysis and segment inspection until it finishes or is stopped. This is
intentional: the native StreamMind runner swaps out ordinary `llama-server` and
holds recurrent state. On an orchestrator restart, in-progress jobs are marked
failed rather than silently resumed.

Offline analysis asks Mage for structured JSON. If model output is not valid
JSON, the raw response is preserved in the event instead of being discarded or
presented as a fabricated timeline.

### What CI and tests prove

`tests/test_mcp_orchestrator.py` exercises the path boundary, SQLite event
cursor, and queued cancellation behavior. GitHub Actions additionally checks
the pinned native patch applies and compiles the CPU `llama-mage-codec-stream`
target. These checks do **not** prove a real CUDA container, a model download,
or end-to-end video inference; run `setup-runtime` and a local video task for
that validation.

See [CONTEXT.md](CONTEXT.md) for the domain vocabulary and
[`docs/adr`](docs/adr) for the two architectural decisions.

## Native StreamMind gate

This fork executes Microsoft's proactive StreamMind path entirely inside
llama.cpp: Mage-ViT embeddings are grouped by codec timestamp, averaged over
patches, passed through the stateful Mamba-1 EPFE, and scored by the four-layer
Qwen3 gate classifier. It does not reconstruct Transformers tensors or keep a
second BF16 model in VRAM.

```bash
cmake -S llama.cpp -B llama.cpp/build -DGGML_VULKAN=ON -DLLAMA_BUILD_EXAMPLES=ON
cmake --build llama.cpp/build --target llama-streammind-e2e -j
GGML_VK_VISIBLE_DEVICES=0 llama.cpp/build/bin/llama-streammind-e2e \
  models/mage-vl-backbone-Q8_0.gguf models/mage-vit-mmproj-Q8_0.gguf \
  models/mage-streammind-epfe-Q8_0.gguf models/mage-streammind-cls-Q8_0.gguf \
  video.mcv
```

Each JSONL row contains the source frame, official silent/speak logits, speak
probability, and the raw decision at Microsoft's `0.5` boundary. The gate is
application-neutral: clients decide what a `speak` event means and may apply
their own policy. `STREAMMIND_CHUNK=N` processes input incrementally while
preserving recurrent state.

### MP4, RTSP and HLS input

`streammind_native.py` is a transport/preprocessing adapter. In incremental
mode one persistent FFmpeg process decodes the stream and the open readiness
selector builds Mage canvases as evidence becomes sufficient; neither executes a
neural model. Mage-ViT, the Mamba-1 EPFE and gate classifier all execute in the
patched C++ llama.cpp runtime, so no Transformers checkpoint or BF16 duplicate
is loaded.

Install only the video preprocessor environment, then keep it active so
`codec-video-prep` and `cv-preinfer` are on `PATH`:

```bash
python3.12 -m venv .venv-codec
source .venv-codec/bin/activate
pip install "codec-video-prep>=0.2.5"
```

Local MP4:

```bash
python tools/streammind_native.py video.mp4 \
  --runner llama.cpp/build/bin/llama-streammind-e2e \
  --backbone models/mage-vl-backbone-Q8_0.gguf \
  --mmproj models/mage-vit-mmproj-Q8_0.gguf \
  --epfe models/mage-streammind-epfe-Q8_0.gguf \
  --classifier models/mage-streammind-cls-Q8_0.gguf \
  --incremental-producer tools/live_codec_stream.py \
  --vulkan-device 0
```

RTSP camera and HLS use the same command; only the source changes:

```bash
python tools/streammind_native.py 'rtsp://user:password@camera/stream1' ...
python tools/streammind_native.py 'https://host/live/playlist.m3u8' ...
```

Incremental live mode has no fixed transport segment and writes no temporary
MP4. At the default 8 sampled FPS it can close after the minimum eight samples
(one second) when readiness and temporal coverage are satisfied; otherwise it
extends up to `--sampled-frames`. EPFE state continues across every adaptive
group until the process exits. For local files, `--realtime` makes FFmpeg feed
frames at playback speed. Small MAGECV1 handoff bundles are deleted immediately
after consumption. Omitting `--incremental-producer` retains the segmented
codec-bitcost compatibility path for offline/reference work.

Repeatable conversion files and local-inference instructions for
[`microsoft/Mage-VL`](https://huggingface.co/microsoft/Mage-VL). The released
GGUF weights were measured on image, video and language benchmarks and compared
with Microsoft's reported BF16 reference values.

**Model weights:** [JohnTdi/Mage-VL-GGUF on Hugging Face](https://huggingface.co/JohnTdi/Mage-VL-GGUF)

![Mage-VL Studio analyzing a selected video range with OCR, runtime metrics and full-frame highlights](docs/mage-vl-studio.png)

*Mage-VL Studio: native Q8 GGUF analysis with a selected time range, dedicated
static-text OCR, RAM/VRAM metrics and representative full-frame Highlights.*

**Development disclosure:** code assistance and review were provided by
OpenAI GPT-5.6 Sol. Final integration, testing and release decisions were made
and verified by the repository maintainer.

This GitHub repository contains the Docker runtime, patches and launch
instructions. The Hugging Face repository contains the Q4/Q8 backbone and
F16/Q8 vision GGUF artifacts.

## Status

| Component | Status |
|---|---|
| Qwen3 language backbone GGUF in patched llama.cpp | Working: Vulkan, CUDA and CPU |
| Mage-ViT `mmproj` conversion to F16/Q8 | Working and quality-validated |
| Native Mage-ViT image/video inference | Working in the included llama.cpp patch |
| Native stateful StreamMind live inference | Working with the included Q8 sidecars |

The Docker images apply a small native runtime patch to pinned llama.cpp. Both
the language backbone and Mage-ViT remain in their GGUF storage types during
inference; no Transformers process or BF16 reconstruction is involved.

## Released variants

| File | Approximate size | Recommended use |
|---|---:|---|
| `mage-vl-backbone-Q8_0.gguf` | 4.69 GB | Best quality GGUF backbone |
| `mage-vl-backbone-Q4_K_M.gguf` | 2.72 GB | Smaller and faster generation |
| `mage-vit-mmproj-Q8_0.gguf` | 353 MB | Compact vision weights |
| `mage-vit-mmproj-F16.gguf` | 661 MB | Maximum vision fidelity |
| `mage-streammind-epfe-Q8_0.gguf` | 96.5 MB | Stateful live-stream memory |
| `mage-streammind-cls-Q8_0.gguf` | 512.6 MB | Silent/speak gate classifier |
| `mage-dcvc-rt-intra-F16.gguf` | 91.3 MB | First/reset frame codec graph |
| `mage-dcvc-rt-inter-F16.gguf` | 41.4 MB | Stateful inter-frame codec graph |

Weights are intentionally not stored in Git. All eight runtime artifacts are in
the [JohnTdi/Mage-VL-GGUF](https://huggingface.co/JohnTdi/Mage-VL-GGUF)
model repository.

## Quick start: native llama.cpp server

### One-command guided installation

After cloning the repository, the installer detects CUDA or Vulkan, estimates
VRAM, selects an 8/16/24–32 GB profile, downloads only the required GGUF files,
derives the matching DRM nodes/groups and starts Docker:

```bash
./install.sh
```

Override detection with `MAGE_BACKEND=vulkan|cuda` and
`MAGE_PROFILE=8|16|24|32`. The generated `.env` remains editable.

The included images compile a pinned llama.cpp revision, apply the Mage-ViT
runtime patch, and include the image/video dependencies. The unused upstream
llama.cpp Web UI is disabled at build time; this avoids Node/npm and mutable UI
downloads, while the gateway provides its own local upload page.

This repository is the complete runtime distribution: Docker clones pinned
llama.cpp, applies the unified native Mage patch from `patches/`, compiles it, and
starts `llama-server` with both GGUF files. A separate llama.cpp fork checkout
is not required.

Requirements: Linux, Git, Python 3 with `venv` and `pip`, Docker Engine, and
Docker Compose 2.30 or newer. Start from an empty directory:

```bash
git clone https://github.com/JohnTDI-cpu/mage-vl-gguf.git
cd mage-vl-gguf

python3 -m venv .hf-venv
.hf-venv/bin/pip install "huggingface_hub>=0.34"
.hf-venv/bin/hf download JohnTdi/Mage-VL-GGUF \
  mage-vl-backbone-Q8_0.gguf mage-vit-mmproj-Q8_0.gguf \
  mage-streammind-epfe-Q8_0.gguf mage-streammind-cls-Q8_0.gguf \
  mage-dcvc-rt-intra-F16.gguf mage-dcvc-rt-inter-F16.gguf \
  --local-dir models

cp .env.example .env
# RADV needs both DRM nodes from the same GPU. Keep their host names unchanged.
sed -i "s/^RENDER_GID=.*/RENDER_GID=$(stat -c '%g' /dev/dri/renderD128)/" .env
sed -i "s/^VIDEO_GID=.*/VIDEO_GID=$(stat -c '%g' /dev/dri/card0)/" .env
docker compose --profile vulkan up -d --build --wait vulkan
curl --fail http://localhost:8080/health
```

The first build compiles our patched llama.cpp and may take several minutes.
When `/health` succeeds, open <http://localhost:8080> in a browser, choose an
MP4, enter a question and click **Analyze video**. No manual conversion or
codec command is required.

For scripts, upload a JPEG/PNG image:

```bash
curl --fail http://localhost:8080/v1/image/analyze \
  -F image=@./your-image.jpg \
  -F 'prompt=Is there a person in this image? Answer yes or no.' \
  -F max_tokens=32
```

Or upload an ordinary MP4. H.264 and HEVC go directly to the official
codec-aware preprocessor; AV1, VP9, MPEG-4 Part 2 and other FFmpeg-readable
video codecs are converted to high-quality H.264 automatically. The container
then packs MAGECV1 and runs native GGUF inference:

```bash
curl --fail http://localhost:8080/v1/video/analyze \
  -F video=@./your-video.mp4 \
  -F 'prompt=Describe the important events in temporal order.' \
  -F max_tokens=256
```

### Continuous live monitoring

Open Mage-VL Studio, select **Live stream**, paste an RTSP/RTMP, direct HTTP/HLS,
localhost or supported page URL such as YouTube, configure the response policy,
and select **Start live analysis**. The native C++ process performs FFmpeg
decode, temporal sampling, DCVC-RT GGUF and Mage canvas construction, then
preserves StreamMind's recurrent EPFE state across groups. Mage-ViT encodes a group once;
the same embeddings feed the gate and any triggered Qwen response. Results with
timestamps appear immediately below the player.

![Mage-VL Studio analyzing a live stream with timestamped responses](docs/mage-vl-live-stream.png)

*Demonstration using a randomly selected public YouTube live stream; the source
was chosen only to exercise the live-analysis path and is not an endorsement.*

The continuous path creates no transport MP4 or MAGECV1 handoff files. Stop the live session before changing
model settings. Source and processing latency depend on network and frame content.
The UI reports live delay, p95 real-time factor (RTF), pending/dropped windows
and whether the stream is keeping pace. **Drop stale windows** is enabled by
default so an overloaded installation stays current instead of analyzing an
ever-growing history.

Live FPS is currently explicit, not automatically benchmarked or adapted to
the user's GPU. The **Analyzed FPS** field defaults to `8`; the hardware profile
sets a conservative starting point, but it does not change FPS while a session
is running. If the pipeline falls behind, the bounded queue drops stale windows
when **Drop stale windows** is enabled instead of accumulating unlimited delay.
Use the reported RTF and queue telemetry to tune it: keep p95 RTF below `0.8`,
reduce analyzed FPS (`12 -> 8 -> 6 -> 4 -> 2`) first, and then lower
`MAGE_DCVC_LIVE_MAX_HEIGHT` (`720 -> 480 -> 360`) if necessary. Increase either
setting only after the stream remains stable for several minutes.

Measured R9700 capacity results are in
[docs/live-performance-r9700.md](docs/live-performance-r9700.md). On this GPU,
the safe default accepts a 1080p/4K source but downsizes it to 480p and samples
8 fps. Native DCVC measured 14.30 fps at 854x480, 6.34 fps at 720p and 1.55 fps
at 1080p. The first empty-cache start can spend 15-17 seconds compiling Vulkan
graphs; warmed sessions do not repeat that cost.

Stop the service with `docker compose --profile vulkan down`. Subsequent starts
can omit `--build`:

```bash
docker compose --profile vulkan up -d --wait vulkan
```

### Choose a quantization

Set the pair in `.env`; all four combinations are supported:

```dotenv
# Highest GGUF quality
GGUF_FILE=mage-vl-backbone-Q8_0.gguf
MMPROJ_FILE=mage-vit-mmproj-F16.gguf

# Recommended compact setup
# GGUF_FILE=mage-vl-backbone-Q4_K_M.gguf
# MMPROJ_FILE=mage-vit-mmproj-Q8_0.gguf
```

Download every variant if you want to switch without downloading later:

```bash
.hf-venv/bin/hf download JohnTdi/Mage-VL-GGUF \
  mage-vl-backbone-Q8_0.gguf mage-vl-backbone-Q4_K_M.gguf \
  mage-vit-mmproj-Q8_0.gguf mage-vit-mmproj-F16.gguf \
  mage-streammind-epfe-Q8_0.gguf mage-streammind-cls-Q8_0.gguf \
  mage-dcvc-rt-intra-F16.gguf mage-dcvc-rt-inter-F16.gguf \
  --local-dir models
```

The API binds to `127.0.0.1` by default because llama-server has no
authentication in this configuration. To expose it deliberately, set
`HOST_BIND` in `.env` and protect it with a firewall or authenticated reverse
proxy. Never forward the raw port directly to the Internet.

The default configuration exposes only the matching
`/dev/dri/renderD128` + `/dev/dri/card0` pair to the container, so another
Vulkan GPU is not visible. Verify both against `/dev/dri/by-path` before first
use. If you change them, also derive `RENDER_GID` and `VIDEO_GID` from the same
nodes. Do not remap them to different names inside the container: RADV follows
their sysfs relationship and can fail authentication when names are rewritten.
An NVIDIA image is available with `--profile cuda`; it needs the NVIDIA
Container Toolkit. See [docker/README.md](docker/README.md).

The public gateway exposes only the local upload page, `/health`,
`/v1/image/analyze`, `/v1/video/prepare`, `/v1/video/analyze-prepared`,
`/v1/video/analyze`, the `/v1/live/sessions` session API and read-only
codec-canvas previews; the internal llama-server listens
only inside the container. Preprocessed videos are cached by content hash and
all output-affecting preprocessing settings, so a repeated upload skips both
transcoding and codec preprocessing. Video audio tracks are deliberately
ignored: Mage-VL analyzes visual content, not speech or sound.

The Live Stream panel accepts RTSP/RTMP, direct HTTP/HLS, localhost URLs and
supported web pages such as YouTube (resolved inside the container with
`yt-dlp`). It offers four response policies: periodic, every detected change,
important changes only, and important changes plus a periodic report. Window
length, minimum response gap, importance sensitivity, visual quality, response
length, RTSP transport and the user prompt are configurable. Closed transport
segments and their MAGECV1 workspaces are deleted immediately after inference;
only the bounded in-memory result history remains. Stop a session explicitly
before changing model settings.

The optimized native live runtime encodes every video window with Mage-ViT
exactly once. Its embeddings feed both StreamMind EPFE/classification and, only
after a trigger, Qwen generation. Starting live mode unloads the ordinary
upload `llama-server`; stopping live restores it, so two language backbones are
never resident simultaneously. Each live result exposes
`vision_encode_count=1` and `shared_vision_embeddings=true` as a runtime check.
The Docker release includes the complete native StreamMind patch and both Q8
EPFE/classifier sidecars are downloaded from the linked Hugging Face repository.

The browser panel previews the local MP4 immediately, shows preprocessing and
inference progress, renders the model answer and PP/TG metrics, and displays
the exact codec canvases passed into Mage-ViT. A canvas is a spatial mosaic of
selected source-frame patches, not necessarily a conventional full frame.
Every card therefore reports its exact source-frame timestamp range and the
complete timestamp list derived from `src_patch_position.npy`.
The player includes a two-handle analysis-range selector. Only the selected
interval is decoded and cached, while every preview timestamp is translated
back to the absolute timeline of the original video.
Its five-step Speed/Detail control changes both temporal sampling (96–320
frames) and the codec-canvas pixel budget (90k–180k). The optional thorough
text scan deliberately runs a separate OCR-focused inference pass: experiments
showed that a single general event prompt can omit a readable static caption
even at the highest visual-detail setting.

Advanced settings can reload the model with a different context, batch,
micro-batch and F16/Q8/Q4 KV cache after an explicit confirmation. If the new
configuration cannot start, the gateway attempts to restore the previous one.
The resource panel reports combined gateway+llama resident RAM on Linux. On
NVIDIA it uses per-process `nvidia-smi` VRAM; on AMD/Vulkan, where the kernel
does not expose reliable per-process VRAM, it reports the selected DRM device's
increase over the pre-load baseline and labels that method explicitly.

### Defaults for a 16 GiB GPU

The shipped profile uses one request slot, F16 KV cache and `ctx=16384`. On a
Radeon AI PRO R9700 with Vulkan, the 50.64-second 1080x1920 H.264 test clip
described in the benchmark repository produced 8,099 prompt tokens:

| Backbone + vision | Peak VRAM | Video prefill | Decode | Request wall |
|---|---:|---:|---:|---:|
| Q8 + Q8 | 7.56 GiB | 2,691 tok/s | 80.66 tok/s | 4.16 s |

The runtime prefill patch combines successive timestamp-text and visual spans
until the decoder batch is full. Before this patch, the same input created many
small Vulkan submissions and reached only 1,221 tok/s. With the patch and
`ctx=32768` it reached 2,718 tok/s and used about 10.4 GiB; reducing the
preallocated context to 16k retains the same F16 KV values while saving roughly
2.8 GiB. Exact values depend on driver, prompt, canvas count and power state.

Safety/resource defaults live in `.env`: 16,384 context tokens, one concurrent
preprocessor, 2 GiB upload, 60-minute duration, 3840x2160 source video, 256
sampled frames, 150,000 pixels per video canvas, 256 generated tokens, and a
15-minute preprocessing timeout. JPEG/PNG images are automatically reduced to
at most 1,048,576 pixels to keep image and video workloads within the 16 GiB
profile. The content-addressed preprocessing cache is capped at 50 GiB and
evicts least-recently-used entries. Change the corresponding `MAGE_*` values in `.env`, then recreate the
service with `docker compose --profile vulkan up -d --force-recreate vulkan`.

The most useful tuning variables are `LLAMA_ARG_CTX_SIZE` (context/VRAM),
`MAGE_SAMPLED_FRAMES` (temporal coverage), `MAGE_MAX_PIXELS` (tokens per codec
canvas), `MAGE_IMAGE_MAX_PIXELS`, and `MAGE_MAX_NEW_TOKENS`. Decoder submission
is explicitly pinned to `MAGE_BATCH_SIZE=2048`, `MAGE_UBATCH_SIZE=512` and F16
KV cache. If a custom image setting creates a visual chunk larger than the
decoder batch, the API returns a clear 422 error instead of accepting a
truncated prompt; increase `MAGE_BATCH_SIZE` or reduce image pixels.

Video duration is not mapped one-to-one to context: the default samples at most
256 frames and readiness grouping produced 24--52 canvases (about 4,800--10,400
visual tokens) across tested 15--85 second clips. More sampled frames or larger
canvases increase preprocessing time, context use and memory. If a request
cannot fit, lower `MAGE_SAMPLED_FRAMES` or `MAGE_MAX_PIXELS`; increase
`LLAMA_ARG_CTX_SIZE` only when sufficient VRAM remains.

## Convert to GGUF

The patch targets llama.cpp commit
`a52077c4cabb4f3c0298329c9d2dd1324d5604cb`. A different revision may require
manual conflict resolution. Run this block from the root of the cloned
`mage-vl-gguf` repository; it creates `llama.cpp/` beneath it.

Use a separate converter venv. Its pinned requirements install CPU PyTorch and
must not replace the ROCm/CUDA environment used for inference.

```bash
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout a52077c4cabb4f3c0298329c9d2dd1324d5604cb
git apply ../patches/llama.cpp-mage-native-streammind.patch
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements-convert_hf_to_gguf.txt

python convert_hf_to_gguf.py ../models/Mage-VL \
  --outfile ../mage-vl-backbone-BF16.gguf --outtype bf16
python convert_hf_to_gguf.py ../models/Mage-VL \
  --mmproj --outfile ../mage-vit-mmproj-F16.gguf --outtype f16
python convert_hf_to_gguf.py ../models/Mage-VL \
  --mmproj --outfile ../mage-vit-mmproj-Q8_0.gguf --outtype q8_0
```

Build llama.cpp and quantize the language backbone:

```bash
cmake -B build -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j
build/bin/llama-quantize ../mage-vl-backbone-BF16.gguf \
  ../mage-vl-backbone-Q8_0.gguf Q8_0
build/bin/llama-quantize ../mage-vl-backbone-BF16.gguf \
  ../mage-vl-backbone-Q4_K_M.gguf Q4_K_M
```

## llama.cpp backbone performance

Radeon AI PRO R9700, Vulkan, llama.cpp `a52077c`, batch 2048, ubatch 512,
Flash Attention enabled:

| Backbone | pp1024 | tg128 |
|---|---:|---:|
| BF16 | 1,380 tok/s | 70.84 tok/s |
| Q8_0 | 5,751 tok/s | 118.03 tok/s |
| Q4_K_M | 5,482 tok/s | 178.41 tok/s |

These figures measure the Qwen3 language backbone, not codec preprocessing or
Mage-ViT. Our released GGUF variants were measured on the complete MMBench EN
dev set (4,329 records), Video-MME tc32 without subtitles (2,700 questions),
WikiText-2 and a nine-image numerical vision comparison. Q8 + vision Q8 scored
84.36% MMBench CircularEval and 63.33% Video-MME. Microsoft's reported BF16
reference values are 84.19% and 64.00%, respectively. The Hugging Face model
card contains the complete comparison, protocol summary and checksums.

## Native validation

The current Vulkan Docker build passes 10 images plus 10 H.264 videos for each
release combination: Q4+vision Q8, Q8+vision Q8, Q4+vision F16 and Q8+vision
F16 — 80/80 deterministic semantic checks. The reusable harness is
`tests/native_sanity.sh`. Gateway tests also cover image and cached-video
inference for all four pairs, AV1-in-MP4 conversion, malformed MAGECV1
rejection, cache reuse, health propagation, graceful shutdown and native live
stream processing. These execution checks complement MMBench and Video-MME.
The release patch chain is compiled on every push by
`.github/workflows/ci.yml`; GPU quality/performance checks remain release-gate
tests because hosted CI has no suitable Vulkan/CUDA device or model weights.

## License and upstream projects

Mage-VL is Apache-2.0 licensed. llama.cpp is MIT licensed. This repository
contains integration patches and documentation; the upstream licenses continue
to apply to their respective code and model artifacts. The [NOTICE](NOTICE)
separates community code, upstream runtime and model terms.