Skip to main content
Glama

🖼️ vision-mcp

Self-hosted multimodal VLM image recognition MCP server

Paste an image in the TUI terminal → the AI client automatically recognizes and returns results · data never leaves your intranet

MCP TypeScript Node Tests Build License: MIT Transport

Claude Code · Codex · OpenCode · any MCP-compatible client


✨ Why use it

Advantage

Description

🔒

Private deployment, data stays on-prem

Connects directly to your self-hosted VLM; images never pass through third-party clouds

🔌

OpenAI-compatible, swappable backend

vLLM / Ollama / GLM-4V / Qwen-VL — pick any, just change the base URL, no code changes

🖼️

Paste-to-recognize in TUI

Paste an image in the terminal, the client automatically calls the tool to recognize it, matching the Zhipu image recognition MCP experience

🧩

Four dedicated tools

General understanding / OCR / diagram understanding / UI-to-code, each with preset system prompts and structured output

📥

Three image input modes

Local path · http(s) URL · data: URI — whatever the client sends, it accepts

🛡️

No error leakage

Error strings contain only static text/status codes; VLM response bodies or stack traces are never leaked to the client

Lightweight single process

stdio; the client spawns the subprocess on demand — no daemon, no server-side state

🔁

Built-in resilience

Auto-retry once on 5xx/timeout, no retry on 4xx, request timeout, image size limit

Full TDD coverage

35 tests + end-to-end round trip (fake VLM + InMemoryTransport)

Related MCP server: openai-vision-mcp-server

📐 Architecture

flowchart LR
    A["🖥️ TUI 客户端<br/>(Claude Code / Codex / OpenCode)"] -- stdio JSON-RPC --> B
    subgraph B["vision-mcp (Node, stdio)"]
        direction TB
        C["tools ×4<br/>analyze_image / extract_text /<br/>understand_diagram / ui_to_code"]
        C --> D["analyze()<br/>共享核心"]
        D --> E["imageSource<br/>路径/URL/data-URI → 归一化"]
        D --> F["vlmClient<br/>OpenAI 兼容 + 重试"]
    end
    F -- HTTPS chat/completions --> G["🧠 自托管 VLM<br/>(qwen-vl / glm-4v / ...)"]
    G -- JSON --> B
    B -- tool result --> A

🛠️ Tools

All share the image_source parameter (local path | http(s) URL | data: URI).

Tool

Tool-specific parameters

Output

analyze_image

prompt (required)

Natural language description / Q&A

extract_text

prompt?, programming_language?

OCR text (code screenshots annotated with language)

understand_diagram

diagram_type? (omit or auto), prompt?

Structured description + mermaid/markdown reproduction

ui_to_code

output_type (code/spec/description), framework? (html/react-tailwind), prompt?

Corresponding code/spec/description

🚀 Quick start

Clone and build

git clone https://github.com/skyone123/vision-mcp.git
cd vision-mcp
npm install
npm run build      # 产出 dist/index.js + dist/index.d.ts
npm test           # 可选:35/35 测试

The client only uses dist/index.jsnote down its absolute path (referred to as $DIST below); you'll need it in the config.

Example: Linux/macOS /home/you/vision-mcp/dist/index.js; Windows D:/git/vision-mcp/dist/index.js.

Environment variables

Variable

Default

Required

Description

VLM_BASE_URL

OpenAI-compatible base, e.g. http://localhost:8000/v1 (include /v1)

VLM_MODEL

qwen-vl-max

Model name

VLM_API_KEY

""

Bearer token; only needed if the backend requires auth — leave empty to omit the Authorization header

VLM_TIMEOUT_MS

60000

Per-request timeout

VLM_MAX_IMAGE_BYTES

10485760

Image size limit 10MB

VLM_MAX_TOKENS

2048

Max tokens in the response

If VLM_BASE_URL is missing, the server exits with an error on startup — no silent failure.

🔧 Configuration

Step 1 · Determine whether the backend needs an API key

curl http://localhost:8000/v1/models
  • 200 + model list → no key needed

  • 401/403key required; retry with the key: curl http://localhost:8000/v1/models -H "Authorization: Bearer your-token"

Pick a vision model from the returned list:

curl -s http://localhost:8000/v1/models | grep '"id"'

Verify it can actually process images (the critical part):

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer 你的token" \
  -d '{
    "model": "qwen-vl-max",
    "messages": [{"role":"user","content":[
      {"type":"text","text":"一句话描述这张图"},
      {"type":"image_url","image_url":{"url":"https://upload.wikimedia.org/wikipedia/commons/thumb/4/47/PNG_transparency_demonstration_1.png/640px-PNG_transparency_demonstration_1.png"}}
    ]}]
  }'

If it returns normal text → the endpoint works; copy those values into env.

Step 2 · Add it to your client

Replace $DIST below with the absolute path to dist/index.js you noted earlier; use node as the command.

claude mcp add vision-mcp --scope user \
  --env VLM_BASE_URL=http://localhost:8000/v1 \
  --env VLM_MODEL=qwen-vl-max \
  -- node "$DIST"

If a key is required, add another line: --env VLM_API_KEY=your-token.

{
  "command": "node",
  "args": ["/absolute/path/to/vision-mcp/dist/index.js"],
  "env": {
    "VLM_BASE_URL": "http://localhost:8000/v1",
    "VLM_MODEL": "qwen-vl-max"
  }
}

With a key, add "VLM_API_KEY": "your-token" to env.

{
  "mcpServers": {
    "vision-mcp": {
      "command": "node",
      "args": ["/absolute/path/to/vision-mcp/dist/index.js"],
      "env": { "VLM_BASE_URL": "http://localhost:8000/v1", "VLM_MODEL": "qwen-vl-max" }
    }
  }
}
[mcp_servers.vision-mcp]
command = "node"
args = ["/absolute/path/to/vision-mcp/dist/index.js"]
env = { VLM_BASE_URL = "http://localhost:8000/v1", VLM_MODEL = "qwen-vl-max" }
{
  "mcp": {
    "vision-mcp": {
      "type": "local",
      "command": ["node", "/absolute/path/to/vision-mcp/dist/index.js"],
      "environment": {
        "VLM_BASE_URL": "http://localhost:8000/v1",
        "VLM_MODEL": "qwen-vl-max"
      }
    }
  }
}

Field names may vary slightly between OpenCode versions; if the tool doesn't show up, check its official MCP docs.

Step 3 · Verify

claude mcp list          # 应看到 vision-mcp,状态 connected

The MCP server doesn't need to be manually kept running — the client spawns the subprocess on demand. Then paste an image in the conversation and ask "what's in this image" — the client automatically calls analyze_image; or do it explicitly:

Use the analyze_image tool to look at this image:

💻 Development

npm run dev              # tsx 直接跑源码(开发期)
npm run build            # tsup 打包 dist/index.js
npm test                 # vitest,35/35
npx tsc --noEmit         # 类型检查

Source structure:

src/
  config.ts          # env → VlmConfig
  imageSource.ts     # loadImage: 路径/URL/data-URI 归一化
  vlmClient.ts       # complete: 调 OpenAI 兼容端点 + 重试/超时
  analyze.ts         # 共享核心: loadImage + complete
  server.ts          # McpServer 注册 + stdio + main
  index.ts           # #!/usr/bin/env node 入口
  tools/
    analyzeImage.ts
    extractText.ts
    understandDiagram.ts
    uiToCode.ts

Each file has a single responsibility and can be tested independently; the four tools are thin wrappers around analyze(), each baking in its own system prompt.

🗺️ Roadmap (optional extensions)

Current scope: stdio only · single backend · single image · no persistence. The following are on-demand extensions:

Candidate

Value

Recommendation

Streaming output

ui_to_code output can be long; streaming lets you see it as it's generated

👍 Worth doing, better UX

Image preprocessing

Scale/compress by long edge before sending — saves tokens, reduces timeouts

👍 Worth doing, lower cost

Structured output

extract_text/understand_diagram return JSON

🤔 Depends on the use case

HTTP/SSE transport

Shared by multiple clients, remote deployment

🤔 stdio is enough for now; on demand

Multi-backend routing

Route different tasks to different VLMs

❌ YAGNI

Video/multi-image batch processing

❌ Beyond current scope

Server-side caching

Cache recognition of identical images

❌ YAGNI

📄 License

MIT © 2026 luyuxin


Related MCP Connectors

Related MCP Servers