vision-mcp
Allows using a self-hosted Ollama instance with a vision model to perform image understanding, OCR, diagram analysis, and UI-to-code conversion.
Supports any OpenAI-compatible chat completions endpoint, enabling the server to connect to compatible vision models, including OpenAI's API, for image recognition tasks.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@vision-mcpExtract text from this screenshot"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
🖼️ vision-mcp
Self-hosted multimodal VLM image recognition MCP server
Paste an image in the TUI terminal → the AI client automatically recognizes and returns results · data never leaves your intranet
Claude Code · Codex · OpenCode · any MCP-compatible client
✨ Why use it
Advantage | Description | |
🔒 | Private deployment, data stays on-prem | Connects directly to your self-hosted VLM; images never pass through third-party clouds |
🔌 | OpenAI-compatible, swappable backend | vLLM / Ollama / GLM-4V / Qwen-VL — pick any, just change the base URL, no code changes |
🖼️ | Paste-to-recognize in TUI | Paste an image in the terminal, the client automatically calls the tool to recognize it, matching the Zhipu image recognition MCP experience |
🧩 | Four dedicated tools | General understanding / OCR / diagram understanding / UI-to-code, each with preset system prompts and structured output |
📥 | Three image input modes | Local path · http(s) URL · |
🛡️ | No error leakage | Error strings contain only static text/status codes; VLM response bodies or stack traces are never leaked to the client |
⚡ | Lightweight single process | stdio; the client spawns the subprocess on demand — no daemon, no server-side state |
🔁 | Built-in resilience | Auto-retry once on 5xx/timeout, no retry on 4xx, request timeout, image size limit |
✅ | Full TDD coverage | 35 tests + end-to-end round trip (fake VLM + InMemoryTransport) |
Related MCP server: openai-vision-mcp-server
📐 Architecture
flowchart LR
A["🖥️ TUI 客户端<br/>(Claude Code / Codex / OpenCode)"] -- stdio JSON-RPC --> B
subgraph B["vision-mcp (Node, stdio)"]
direction TB
C["tools ×4<br/>analyze_image / extract_text /<br/>understand_diagram / ui_to_code"]
C --> D["analyze()<br/>共享核心"]
D --> E["imageSource<br/>路径/URL/data-URI → 归一化"]
D --> F["vlmClient<br/>OpenAI 兼容 + 重试"]
end
F -- HTTPS chat/completions --> G["🧠 自托管 VLM<br/>(qwen-vl / glm-4v / ...)"]
G -- JSON --> B
B -- tool result --> A🛠️ Tools
All share the image_source parameter (local path | http(s) URL | data: URI).
Tool | Tool-specific parameters | Output |
|
| Natural language description / Q&A |
|
| OCR text (code screenshots annotated with language) |
|
| Structured description + mermaid/markdown reproduction |
|
| Corresponding code/spec/description |
🚀 Quick start
Clone and build
git clone https://github.com/skyone123/vision-mcp.git
cd vision-mcp
npm install
npm run build # 产出 dist/index.js + dist/index.d.ts
npm test # 可选:35/35 测试The client only uses dist/index.js — note down its absolute path (referred to as $DIST below); you'll need it in the config.
Example: Linux/macOS
/home/you/vision-mcp/dist/index.js; WindowsD:/git/vision-mcp/dist/index.js.
Environment variables
Variable | Default | Required | Description |
| — | ✅ | OpenAI-compatible base, e.g. |
|
| — | Model name |
|
| — | Bearer token; only needed if the backend requires auth — leave empty to omit the |
|
| — | Per-request timeout |
|
| — | Image size limit 10MB |
|
| — | Max tokens in the response |
If
VLM_BASE_URLis missing, the server exits with an error on startup — no silent failure.
🔧 Configuration
Step 1 · Determine whether the backend needs an API key
curl http://localhost:8000/v1/models200+ model list → no key needed401/403→ key required; retry with the key:curl http://localhost:8000/v1/models -H "Authorization: Bearer your-token"
Pick a vision model from the returned list:
curl -s http://localhost:8000/v1/models | grep '"id"'Verify it can actually process images (the critical part):
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer 你的token" \
-d '{
"model": "qwen-vl-max",
"messages": [{"role":"user","content":[
{"type":"text","text":"一句话描述这张图"},
{"type":"image_url","image_url":{"url":"https://upload.wikimedia.org/wikipedia/commons/thumb/4/47/PNG_transparency_demonstration_1.png/640px-PNG_transparency_demonstration_1.png"}}
]}]
}'If it returns normal text → the endpoint works; copy those values into env.
Step 2 · Add it to your client
Replace
$DISTbelow with the absolute path todist/index.jsyou noted earlier; usenodeas thecommand.
claude mcp add vision-mcp --scope user \
--env VLM_BASE_URL=http://localhost:8000/v1 \
--env VLM_MODEL=qwen-vl-max \
-- node "$DIST"If a key is required, add another line: --env VLM_API_KEY=your-token.
{
"command": "node",
"args": ["/absolute/path/to/vision-mcp/dist/index.js"],
"env": {
"VLM_BASE_URL": "http://localhost:8000/v1",
"VLM_MODEL": "qwen-vl-max"
}
}With a key, add "VLM_API_KEY": "your-token" to env.
{
"mcpServers": {
"vision-mcp": {
"command": "node",
"args": ["/absolute/path/to/vision-mcp/dist/index.js"],
"env": { "VLM_BASE_URL": "http://localhost:8000/v1", "VLM_MODEL": "qwen-vl-max" }
}
}
}[mcp_servers.vision-mcp]
command = "node"
args = ["/absolute/path/to/vision-mcp/dist/index.js"]
env = { VLM_BASE_URL = "http://localhost:8000/v1", VLM_MODEL = "qwen-vl-max" }{
"mcp": {
"vision-mcp": {
"type": "local",
"command": ["node", "/absolute/path/to/vision-mcp/dist/index.js"],
"environment": {
"VLM_BASE_URL": "http://localhost:8000/v1",
"VLM_MODEL": "qwen-vl-max"
}
}
}
}Field names may vary slightly between OpenCode versions; if the tool doesn't show up, check its official MCP docs.
Step 3 · Verify
claude mcp list # 应看到 vision-mcp,状态 connectedThe MCP server doesn't need to be manually kept running — the client spawns the subprocess on demand. Then paste an image in the conversation and ask "what's in this image" — the client automatically calls analyze_image; or do it explicitly:
Use the analyze_image tool to look at this image:
💻 Development
npm run dev # tsx 直接跑源码(开发期)
npm run build # tsup 打包 dist/index.js
npm test # vitest,35/35
npx tsc --noEmit # 类型检查Source structure:
src/
config.ts # env → VlmConfig
imageSource.ts # loadImage: 路径/URL/data-URI 归一化
vlmClient.ts # complete: 调 OpenAI 兼容端点 + 重试/超时
analyze.ts # 共享核心: loadImage + complete
server.ts # McpServer 注册 + stdio + main
index.ts # #!/usr/bin/env node 入口
tools/
analyzeImage.ts
extractText.ts
understandDiagram.ts
uiToCode.tsEach file has a single responsibility and can be tested independently; the four tools are thin wrappers around analyze(), each baking in its own system prompt.
🗺️ Roadmap (optional extensions)
Current scope: stdio only · single backend · single image · no persistence. The following are on-demand extensions:
Candidate | Value | Recommendation |
Streaming output |
| 👍 Worth doing, better UX |
Image preprocessing | Scale/compress by long edge before sending — saves tokens, reduces timeouts | 👍 Worth doing, lower cost |
Structured output |
| 🤔 Depends on the use case |
HTTP/SSE transport | Shared by multiple clients, remote deployment | 🤔 stdio is enough for now; on demand |
Multi-backend routing | Route different tasks to different VLMs | ❌ YAGNI |
Video/multi-image batch processing | — | ❌ Beyond current scope |
Server-side caching | Cache recognition of identical images | ❌ YAGNI |
📄 License
MIT © 2026 luyuxin
This server cannot be deployed
Maintenance
Related MCP Connectors
OCR, transcription, file extraction, and image generation for AI agents via MCP.
Generate images with any major model — one API key, one prepaid balance, one MCP.
Multi-model AI image and video generator. 14 models behind one OAuth-secured MCP endpoint.
MCP server for Qwen Image 3 AI image generation
Related MCP Servers
- FlicenseNot gradedqualityCmaintenanceEnables AI clients like Claude to understand, analyze, and describe local images via VL models through the MCP protocol.-
- AlicenseNot gradedqualityAmaintenanceEnables image analysis via OpenAI-compatible vision APIs, supporting local files, URLs, and base64 inputs with intelligent tiling for high-resolution images. Provides a secure, configurable MCP stdio server for structured vision analysis.184 npm2MIT
- AlicenseAqualityBmaintenanceEnables any MCP client to perform image understanding and OCR via any OpenAI-compatible vision-language model. Supports local, private inference without images leaving the machine.214 npmMIT
- AlicenseAqualityCmaintenanceProvides image recognition capabilities to MCP clients by integrating with OpenAI-compatible vision models, supporting local images, URLs, multi-image comparison, and model listing.4MIT