How do I use vlm-mcp?

1. Click on "Deploy Server". 2. Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state. 3. In the chat, type @ followed by the MCP server name and your instructions, e.g., "@vlm-mcp analyze this chart image and explain the trend" That's it! The server will respond to your query, and you can continue using it as needed. Here is a step-by-step guide with screenshots.

de en es ja ko ru zh

vlm-mcp

by YC-CLT

Overview Schema Source Related Servers Discussions Admin

Python

Hybrid

VLM-MCP

GitHub License: MIT Python 3.11+ MCP

English | 中文

VLM-based image understanding MCP Server. Supports local llama.cpp and online VLMs (e.g. Qwen3-VL-Flash) via a unified OpenAI-compatible API.

English

Features

Dual backend: local llama.cpp + online Qwen3-VL-Flash, unified OpenAI-compatible API
Three-tier cache: L1 image encoding cache, L2 response cache (with TTL), L3 llama-server KV cache
Session management: multi-turn conversation context, auto-eviction and timeout cleanup
Prompt templates: built-in describe / ocr / chart / translate / qa
Lifecycle management: llama-server subprocess auto-starts/stops with MCP, no manual management
Backend health: auto-disable backends on API key errors, manual enable/disable support
Multi-source images: local path, HTTP URL, Base64 Data URI, raw Base64 fallback

Architecture

MCP Client (stdio)
       │
       ▼
  server.py ── tool layer (ocr_image / analyze_image / create_session / ...)
       │
       ├── session_manager.py ── session lifecycle
       ├── cache.py ── L1 image cache + L2 response cache
       ├── image_utils.py ── image parsing (path/URL/Base64)
       │
       ▼
  providers/ ── OpenAI-compatible + OCR
       │
       ├── openai_compat.py ── llama.cpp (lazy start) / online VLM
       └── ocr_provider.py ── RapidOCR (lazy load, delayed unload)

Quick Start

Requirements

Component	Notes
Python 3.11+	Runtime
uv	Package manager
llama.cpp	Native binary (`llama-server`), CUDA build required
rapidocr-onnxruntime	OCR engine (auto-installed as dependency)
Qwen3-VL-8B GGUF	Language model + vision projector

Note: This project uses the llama.cpp native binary (llama-server), NOT llama-cpp-python. No Python bindings needed — just download the llama.cpp executable.

Recommended model: Download two files from Qwen3-VL-8B-Instruct-GGUF:

File	Recommended	Notes
Vision model	`Qwen3VL-8B-Instruct-Q4_K_M.gguf`	Q4_K_M quantization, balance of speed & accuracy
Vision projector	`mmproj-Qwen3VL-8B-Instruct-F16.gguf`	Must be F16, do not quantize

8 GB VRAM is sufficient. Online-only mode (qwen-vl backend only) can skip llama.cpp and GGUF models.

Install

git clone https://github.com/YC-CLT/VLM-mcp.git
cd VLM-mcp
uv sync

Configure

cp config.example.json config.json

Edit config.json:

{
  "backends": {
    "llama-cpp": {
      "enabled": true,
      "base_url": "http://localhost:11433/v1",
      "api_key": "sk-no-key-required",
      "model_name": "qwen3-vl"
    },
    "qwen-vl": {
      "enabled": false,
      "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
      "api_key": "your-dashscope-api-key",
      "model_name": "qwen-vl-flash"
    }
  },
  "default_backend": "llama-cpp",
  "cache_enabled": true,
  "llama": {
    "server_exe": "llama-server",
    "model": "D:/path/to/Qwen3VL-8B-Instruct-Q4_K_M.gguf",
    "mmproj": "D:/path/to/mmproj-Qwen3VL-8B-Instruct-F16.gguf",
    "ngl": 99,
    "ctx_size": 8192,
    "predict": 16384,
    "temperature": 0.7,
    "top_k": 20,
    "top_p": 0.8,
    "repeat_penalty": 1.0,
    "presence_penalty": 1.0,
    "flash_attn": "auto",
    "threads": 4,
    "batch_size": 512,
    "cache_type_k": "q8_0",
    "cache_type_v": "q8_0",
    "parallel": 1,
    "image_min_tokens": 1024,
    "image_max_tokens": 2048,
    "alias": "qwen3-vl",
    "health_poll_interval": 3,
    "health_poll_timeout": 180
  }
}

Key fields:

backends.<name>.enabled: set false to manually disable a backend
llama.model / llama.mmproj: absolute paths to model files (required)
llama.ngl: GPU layers, 99 = all GPU, 0 = CPU only
llama.server_exe: llama-server executable, defaults to PATH lookup

Run

uv run main.py

llama-server subprocess auto-starts and stops with MCP. No manual management needed.

MCP stdio transport — configure in your MCP client's mcpServers config.

Run from any directory: uv run --directory D:\CodeFile\VLM-mcp main.py

MCP Client Config

Add to your MCP client configuration:

{
  "mcpServers": {
    "vlm-mcp": {
      "type": "stdio",
      "command": "uv",
      "args": ["run", "--directory", "/path/to/VLM-mcp", "main.py"]
    }
  }
}

MCP Tools

Tool	Parameters	Description
`ocr_image`	`image`	OCR text extraction (RapidOCR, offline, ms-level)
`analyze_image`	`image`, `prompt`, `template`, `params`, `backend`, `session_id`	Analyze image with template & session support
`create_session`	`backend`	Create multi-turn conversation session
`close_session`	`session_id`	Close session
`list_sessions`	—	List all active sessions
`list_backends`	—	List backends and their status
`list_templates`	—	List available prompt templates

Templates

Template	Params	Description
`describe`	—	General image description
`ocr`	—	Text extraction
`chart`	—	Chart analysis
`translate`	`target_lang`	Image translation (default: zh)
`qa`	`question`	Image Q&A

Examples

// Single analysis
{
  "tool": "analyze_image",
  "args": {
    "image": "D:/photos/cat.png",
    "prompt": "What is in this image?"
  }
}

// Using template
{
  "tool": "analyze_image",
  "args": {
    "image": "https://example.com/chart.png",
    "template": "chart"
  }
}

// Multi-turn session
{ "tool": "create_session", "args": { "backend": "llama-cpp" } }
// → { "session_id": "xxx" }
{ "tool": "analyze_image", "args": { "image": "...", "prompt": "...", "session_id": "xxx" } }
{ "tool": "analyze_image", "args": { "prompt": "Tell me more", "session_id": "xxx" } }
{ "tool": "close_session", "args": { "session_id": "xxx" } }

Configuration Constants

Non-sensitive constants in config.py:

Constant	Default	Description
`IMAGE_MAX_SIZE_MB`	20	Max image size
`IMAGE_DOWNLOAD_TIMEOUT`	10	Image download timeout (s)
`CACHE_IMAGE_MAX_ENTRIES`	100	L1 cache limit
`CACHE_RESPONSE_MAX_ENTRIES`	500	L2 cache limit
`CACHE_RESPONSE_TTL_ONLINE`	3600	Online backend cache TTL (s)
`CACHE_RESPONSE_TTL_LOCAL`	1800	Local backend cache TTL (s)
`SESSION_TTL`	1800	Session timeout (s)
`SESSION_MAX`	5	Max sessions per backend
`LOG_LEVEL`	"INFO"	Log level

Development

uv sync --dev
uv run pytest tests/ -v

FAQ

llama-server running on CPU?
Check llama.ngl in config.json — 99 = all GPU, 0 = CPU only.

llama-server fails to start?
Verify server_exe is executable and model/mmproj paths exist. Check llama_server.log.

Online backend returns 401?
Invalid API key auto-disables the backend. Set a valid key and restart. Or set "enabled": false to skip.

Port conflict?
MCP uses stdio transport. llama-server uses port 11433 (configurable in config.json).

VLM calls are very slow or timeout?
May be caused by system HTTP proxy. The MCP talks to 127.0.0.1:11433 which should never go through a proxy. Ensure NO_PROXY includes localhost,127.0.0.1. Latest version handles this automatically.

Related MCP server: MCP Vision Server

中文

特性

双后端支持：本地 llama.cpp + 在线 Qwen3-VL-Flash，统一 OpenAI 兼容 API
三层缓存：L1 图片编码缓存、L2 响应缓存（带 TTL）、L3 llama-server KV Cache
会话管理：多轮对话上下文保持，自动淘汰与超时清理
提示词模板：内置 describe / ocr / chart / translate / qa 模板
生命周期管理：llama-server 子进程与 MCP 同起同停，启动即用，无需手动管理
后端健康：API Key 错误自动禁用后端，支持手动启用/禁用
多图片来源：本地路径、HTTP URL、Base64 Data URI、纯 Base64 回退

架构

MCP Client (stdio)
       │
       ▼
  server.py ── 工具层 (ocr_image / analyze_image / create_session / ...)
       │
       ├── session_manager.py ── 会话生命周期
       ├── cache.py ── L1 图片缓存 + L2 响应缓存
       ├── image_utils.py ── 图片解析 (路径/URL/Base64)
       │
       ▼
  providers/ ── OpenAI 兼容 + OCR
       │
       ├── openai_compat.py ── llama.cpp (懒启动) / 在线 VLM
       └── ocr_provider.py ── RapidOCR (懒加载、延迟卸载)

快速开始

环境要求

组件	说明
Python 3.11+	运行环境
uv	包管理
llama.cpp	原生二进制（`llama-server`），需 CUDA 版
Qwen3-VL-8B GGUF	语言模型 + 视觉投影器

注意：本项目使用 llama.cpp 原生二进制（llama-server），不是 llama-cpp-python。无需安装 Python 绑定（即无需llama-cpp-python，这个和单llama.cpp相互独立），只需下载 llama.cpp 可执行文件即可。

推荐模型下载：从 Qwen3-VL-8B-Instruct-GGUF 下载两个文件：

文件	推荐	说明
视觉模型	`Qwen3VL-8B-Instruct-Q4_K_M.gguf`	Q4_K_M 量化，平衡速度与精度
图像编码器	`mmproj-Qwen3VL-8B-Instruct-F16.gguf`	建议 F16，不必量化

这样8G显存就可以跑

纯在线模式（仅用 qwen-vl 后端）可跳过 llama.cpp 和 GGUF 模型。

安装

git clone https://github.com/YC-CLT/VLM-mcp.git
cd VLM-mcp
uv sync

配置

cp config.example.json config.json

编辑 config.json：

{
  "backends": {
    "llama-cpp": {
      "enabled": true,
      "base_url": "http://localhost:11433/v1",
      "api_key": "sk-no-key-required",
      "model_name": "qwen3-vl"
    },
    "qwen-vl": {
      "enabled": false,
      "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
      "api_key": "your-dashscope-api-key",
      "model_name": "qwen-vl-flash"
    }
  },
  "default_backend": "llama-cpp",
  "cache_enabled": true,
  "llama": {
    "server_exe": "llama-server",
    "model": "D:/path/to/Qwen3VL-8B-Instruct-Q4_K_M.gguf",
    "mmproj": "D:/path/to/mmproj-Qwen3VL-8B-Instruct-F16.gguf",
    "ngl": 99,
    "ctx_size": 8192,
    "predict": 16384,
    "temperature": 0.7,
    "top_k": 20,
    "top_p": 0.8,
    "repeat_penalty": 1.0,
    "presence_penalty": 1.0,
    "flash_attn": "auto",
    "threads": 4,
    "batch_size": 512,
    "cache_type_k": "q8_0",
    "cache_type_v": "q8_0",
    "parallel": 1,
    "image_min_tokens": 1024,
    "image_max_tokens": 2048,
    "alias": "qwen3-vl",
    "health_poll_interval": 3,
    "health_poll_timeout": 180
  }
}

关键字段：

backends.<name>.enabled：设为 false 可手动禁用后端
llama.model / llama.mmproj：本地模型文件绝对路径（必填）
llama.ngl：GPU 层数，99 表示全部 offload 到 GPU，0 为纯 CPU
llama.server_exe：llama-server 可执行文件，默认从 PATH 查找

运行

uv run main.py

启动后会自动拉起 llama-server 子进程，MCP 退出时自动停止。无需手动管理 llama-server。

MCP stdio 传输 — 在 MCP 客户端 mcpServers 配置中设置。

从任意目录运行：uv run --directory D:\CodeFile\VLM-mcp main.py

MCP 客户端配置

在你的 MCP 客户端配置文件中添加：

{
  "mcpServers": {
    "vlm-mcp": {
      "type": "stdio",
      "command": "uv",
      "args": ["run", "--directory", "/path/to/VLM-mcp", "main.py"]
    }
  }
}

MCP 工具

工具	参数	说明
`ocr_image`	`image`	OCR 文字提取（RapidOCR，离线毫秒级）
`analyze_image`	`image`, `prompt`, `template`, `params`, `backend`, `session_id`	分析图片，支持模板和会话
`create_session`	`backend`	创建多轮对话会话
`close_session`	`session_id`	关闭会话
`list_sessions`	—	列出所有活跃会话
`list_backends`	—	列出后端及其状态
`list_templates`	—	列出可用提示词模板

模板

模板	参数	说明
`describe`	—	通用图片描述
`ocr`	—	文字提取
`chart`	—	图表分析
`translate`	`target_lang`	图片翻译（默认中文）
`qa`	`question`	图片问答

使用示例

// 单次分析
{
  "tool": "analyze_image",
  "args": {
    "image": "D:/photos/cat.png",
    "prompt": "这张图片里有什么？"
  }
}

// 使用模板
{
  "tool": "analyze_image",
  "args": {
    "image": "https://example.com/chart.png",
    "template": "chart"
  }
}

// 多轮会话
{ "tool": "create_session", "args": { "backend": "llama-cpp" } }
// → { "session_id": "xxx" }
{ "tool": "analyze_image", "args": { "image": "...", "prompt": "...", "session_id": "xxx" } }
{ "tool": "analyze_image", "args": { "prompt": "继续分析", "session_id": "xxx" } }
{ "tool": "close_session", "args": { "session_id": "xxx" } }

配置常量

非敏感常量集中于 config.py，可在代码中直接修改：

常量	默认值	说明
`IMAGE_MAX_SIZE_MB`	20	图片最大体积
`IMAGE_DOWNLOAD_TIMEOUT`	10	图片下载超时（秒）
`CACHE_IMAGE_MAX_ENTRIES`	100	L1 缓存上限
`CACHE_RESPONSE_MAX_ENTRIES`	500	L2 缓存上限
`CACHE_RESPONSE_TTL_ONLINE`	3600	在线后端缓存 TTL（秒）
`CACHE_RESPONSE_TTL_LOCAL`	1800	本地后端缓存 TTL（秒）
`SESSION_TTL`	1800	会话超时（秒）
`SESSION_MAX`	5	每后端最大会话数
`LOG_LEVEL`	"INFO"	日志级别

开发

uv sync --dev
uv run pytest tests/ -v

常见问题

llama-server 跑在 CPU 上？
检查 config.json 中 llama.ngl 是否为 99（全 GPU），0 为纯 CPU。

llama-server 启动失败？
确认 server_exe 可执行（PATH 中或绝对路径），model/mmproj 路径存在。查看 llama_server.log。

在线后端 401 错误？
API Key 无效时会自动禁用该后端，设好 Key 后重启即可恢复。也可手动设 "enabled": false 跳过。

端口被占用？
MCP 使用 stdio 传输。llama-server 端口 11433（可在 config.json 中配置）。

VLM 调用很慢或超时？
可能是系统 HTTP 代理导致。MCP 与 127.0.0.1:11433 通信不应走代理。确保 NO_PROXY 包含 localhost,127.0.0.1。最新版已自动处理。

许可

MIT

This server cannot be deployed

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Maintenance

ActivityMaintained

ResponsivenessNo issues

Resources

GitHub Repository

Need Help?

Related Servers

Related MCP Connectors

APICK AI A
app.apick
LLM chat, text tools, image generation, editing, batch image jobs, and asynchronous video generation
AI & Machine Learning Image & Video Processing
1511
SammaPix — Image & PDF tools for AI agents
io.github.samma1997
Image & PDF tools for AI agents: compress, convert, resize, PDF, AI vision, pipeline.
Image & Video Processing AI & Machine Learning
17
rostro AOAuth
dev.rostro
Turn any LLM multimodal; generate images, voices, videos, 3D models, music, and more.
Multimedia Processing Image & Video Processing Audio Processing
2
Frenchie A
io.github.Lab94
OCR, transcription, file extraction, and image generation for AI agents via MCP.
AI & Machine Learning Image & Video Processing Speech Processing
72

Related MCP Servers

Vision MCP
Image & Video Processing Multimedia Processing
i-richardwang
A
license
A
quality
D
maintenance
Enables image analysis and understanding using Vision Language Models through OpenAI-compatible APIs. Supports analyzing images from URLs or local files with custom prompts.
Updated 10 months ago (2025-12-26 05:40 UTC)
1
2
MIT
MCP Vision Server
Image & Video Processing Autonomous Agents Multimedia Processing
LZMW
A
license
A
quality
D
maintenance
Provides advanced image analysis capabilities including object recognition, OCR text extraction, and multi-turn visual dialogues using OpenAI-compatible APIs. It supports both local files and Base64 inputs with additional features for session persistence and web-based configuration management.
Updated 8 months ago (2026-02-24 01:46 UTC)
3
MIT
api-vision-mcp
Image & Video Processing AI & Machine Learning
stzhiyang
F
license
Not graded
quality
D
maintenance
Enables image recognition using vision models via OpenAI-compatible APIs, supporting multiple platforms like OpenAI, DeepSeek, and Ollama.
Updated 3 months ago (2026-07-09 12:35 UTC)
-
vision-mcp
knownothing20
A
license
Not graded
quality
D
maintenance
Enables AI agents to analyze images using any OpenAI-compatible vision API, providing tools for image analysis, OCR, error diagnosis, diagram understanding, and chart analysis.
Updated 3 months ago (2026-07-25 06:58 UTC)
MIT