Skip to main content
Glama
Tip-Tutaki

media2html

by Tip-Tutaki
OverviewSchemaRelated ServersScoreDiscussions
Python
Local

Media2HTML: A VLM-Free Pure Compiler for Multimodal Agents

Disclaimer: Please understand this is V0.01 and still very experimental. It is the product of a personal theory and nothing more.

A tool that converts images, videos, and audio into structured, cacheable HTML representations optimized for text-based LLM reasoning.

V0.0.1 — Pure Compiler Architecture

Concept

Modern multimodal AI agents need to reason about media files, but LLMs are fundamentally text-based. Traditional approaches — passing raw base64 images or unstructured VLM captions — suffer from massive token overhead, hallucination risk, and GPU contention.

Media2HTML eliminates all of this. It extracts structured facts (objects, text, colors, spatial relations, semantic tags) using deterministic algorithms, then compiles them into semantic HTML that LLMs understand natively. Zero Vision-Language Models in the runtime loop.

Key benefits:

  • (To be dertimined) token reduction vs. raw base64 images

  • 0 GB VRAM — all extraction runs on CPU, leaving GPU free for the text LLM

  • 100% deterministic — same input always produces the same output

  • Zero hallucination — no generative model in the pipeline

  • Instant caching — repeated file analysis returns in <1ms

Related MCP server: HTML to Markdown Converter MCP

V0.0.1 Stack

Component

Technology

Purpose

Object Detection

YOLO-World + SAHI

Open-vocabulary detection with sliced inference for 4K+ images

Z-Axis Layout

Depth-Anything V2

Foreground / midground / background categorization

Semantic Tags

CLIP (openai/clip-vit-base-patch32)

Zero-shot scene tagging (rain, night, cherry blossom, etc.)

Text Extraction

RapidOCR

CPU-only OCR with paragraph grouping

Visual Maps

ASCILINE

Token-optimized 512-cell grid for visual reasoning

Audio

Whisper + Pyannote

Transcription and speaker diarization

Video

scenedetect + FFmpeg

Scene detection with keyframe extraction

Setup

1. Install System Dependencies

sudo apt install ffmpeg portaudio19-dev

2. Install Python Packages

pip install -e .

3. Environment Variables (Optional)

export HF_TOKEN="hf_..."  # Required for Pyannote audio diarization

Usage

As a Python Library

from media2html import media_to_html

# Image
html = media_to_html("path/to/image.jpg", mode="rich")

# Video (with interleaved audio)
html = media_to_html("path/to/video.mp4", mode="compact")

# Audio
html = media_to_html("path/to/audio.wav", mode="minimal")

As an MCP Server

# Start the MCP server
python3 -m media2html.mcp_server

# Or run directly
python3 media2html/mcp_server.py

Agents call the transcode_media tool to analyze any local media file.

Modes

Mode

Content

Token Count

minimal

Caption + top objects + colors

~200

compact

+ OCR + visual grid

~800

rich

+ spatial relations, semantic tags, depth scene, accent colors

~2500

Output Format

Media2HTML produces semantic HTML that leverages LLM pretraining priors:

<image-summary width="1376" height="768" source="photo.png">
  <caption>Scene contains: car, person, umbrella. Text detected: 'LOFUGA'. Dominant color: #303048. Tags: rain, wet road.</caption>
  <spatial-graph>
    <rel>car is left of person</rel>
    <rel>person is above car</rel>
  </spatial-graph>
  <semantic-tags>
    <t>rain</t>
    <t>wet road</t>
  </semantic-tags>
  <scene>
    <foreground>car</foreground>
    <background>person, umbrella</background>
  </scene>
  <accent-colors>
    <c hex="#e0c4c4" pct="0.04" label="vibrant"/>
  </accent-colors>
  <objects>
    <obj label="car" bbox="0.277,0.438,0.758,1.000"/>
  </objects>
  <text-regions>
    <t bbox="0.350,0.750,0.412,0.794">LOFUGA</t>
  </text-regions>
  <colors>
    <c hex="#303048" pct="0.08"/>
  </colors>
</image-summary>

Architecture

media2html/
├── pyproject.toml           # Package configuration
├── requirements.txt         # Dependencies
├── README.md                # This file
├── TECHNICAL_WHITEPAPER.md  # Full architecture documentation
├── test_media2html.py       # Test suite
└── media2html/
    ├── __init__.py          # Package exports
    ├── cache.py             # Disk-based caching (diskcache + MD5)
    ├── html_builder.py      # Data model + HTML generation
    ├── pipeline.py          # Main extraction pipeline
    ├── mcp_server.py        # MCP server for agent integration
    └── extractors/
        ├── __init__.py
        ├── vision.py        # YOLO-World + Depth-Anything + CLIP + RapidOCR
        ├── audio.py         # Whisper + Pyannote
        └── video.py         # Scene detection + keyframes

Performance

Metric

Value

Token reduction vs. raw base64

to be determined

Extraction latency (minimal)

~200ms

Extraction latency (rich)

~700ms

GPU VRAM required

0 GB (all CPU)

Deterministic output

100%

Hallucination risk

0%

Cache hit latency

<1ms

License Creative Commons

This server cannot be installed

F
license - not found
-
quality - not tested
C
maintenance

How are these scores calculated?

Maintenance

–Maintainers
–Response time
–Release cycle
–Releases (12mo)
Commit activity

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/Tip-Tutaki/media2html'

If you have feedback or need assistance with the MCP directory API, please join our Discord server