Vision MCP Server

A Model Context Protocol (MCP) server that gives vision understanding to agents connected to non-multimodal models (DeepSeek, older GPT-4, local small models, etc.): the agent hands an image to the MCP tool, the server calls a vision model, and returns text.

Supports major providers in China and the US plus any OpenAI-compatible endpoint. Official SDKs first, abstraction before implementation, zero-intrusion provider additions.

中文文档见 README.zh-CN.md

Features

• 4 tools: analyze_image / describe_image / ocr_image / list_providers, all returning plain Markdown text

• 13 built-in providers: OpenAI / Anthropic / Google Gemini / Qwen (DashScope) / Zhipu / Doubao (Volcengine) / ERNIE (Qianfan) / StepFun / Ollama / Alibaba Bailian / SiliconFlow / OpenRouter / custom OpenAI-compatible endpoint

• Three image inputs: local path / http(s) URL / base64 (data URI or raw base64), auto-sniffed

• Three-tier fallback chain: official SDK → OpenAI-compatible endpoint → native fetch (see SPEC §1)

• Stateless: every call is independent; images and results are never cached; keys are read from environment variables only

Related MCP server: vision-mcp

Quick start

Option A: npx (published to npm, no repo needed)

npx -y @inferai/vision-mcp

Option B: local build

git clone <repo> && cd vision-mcp
pnpm install
pnpm build
node dist/index.js

MCP configuration examples (stdio)

The server speaks the stdio transport: the MCP client spawns the process and exchanges JSON-RPC messages over stdin/stdout. Configure it wherever your client defines MCP servers:

• Claude Code: project-level .mcp.json or user-level ~/.claude.json (mcpServers key)

• Claude Desktop: claude_desktop_config.json

• Any MCP client (Cursor, self-built agents, etc.): same structure

npx version (available after the package is published):

{
  "mcpServers": {
    "vision-mcp": {
      "command": "npx",
      "args": ["-y", "@inferai/vision-mcp"],
      "env": {
        "OPENAI_API_KEY": "sk-...",
        "DASHSCOPE_API_KEY": "sk-..."
      }
    }
  }
}

Local development (adjust the path; --env-file-if-exists=.env loads .env natively):

{
  "mcpServers": {
    "vision-mcp": {
      "command": "node",
      "args": ["--env-file-if-exists=.env", "/absolute/path/to/vision-mcp/dist/index.js"],
      "env": {
        "OPENAI_API_KEY": "sk-..."
      }
    }
  }
}

With startup arguments (override provider defaults via argv, see below):

{
  "mcpServers": {
    "vision-mcp": {
      "command": "npx",
      "args": [
        "-y",
        "@inferai/vision-mcp",
        "--default-provider=dashscope",
        "--siliconflow-api-key=sk-...",
        "--siliconflow-model=Qwen/Qwen2.5-VL-7B-Instruct"
      ],
      "env": {
        "DASHSCOPE_API_KEY": "sk-..."
      }
    }
  }
}

stdio notes:

• stdout carries the MCP protocol only — the server never prints logs there; diagnostics go to stderr

• the client manages the process lifecycle (spawn on start, kill on exit); no daemon needed

• first npx run downloads the package and may take a few seconds

• env variables can also come from the shell environment if the client inherits it (no env block needed)

Debug with MCP Inspector:

pnpm dlx @modelcontextprotocol/inspector node dist/index.js --xxx-api-key=xxx --xxx2-api-key=xxx

Setting variables

1. MCP config env block (recommended, most reliable across platforms) — write the variables into the env object above

2. .env file (local development) — copy .env.example to .env, fill it in, then node --env-file-if-exists=.env dist/index.js (Node 22 native, no dotenv needed)

3. Shell export — export OPENAI_API_KEY=sk-xxx then run

Providers without keys show as unavailable in list_providers and report the missing variable when called.

Publishing (before npx works)

pnpm publish          # or pnpm release (changeset flow)

Environment variables

Every provider's API_KEY, BASE_URL, and MODEL support environment overrides (convention: <PROVIDER_PREFIX>_API_KEY / <PROVIDER_PREFIX>_BASE_URL / <PROVIDER_PREFIX>_MODEL):

? = optional (has a built-in default); * = required.

Global configuration:

Fallback chain

When multiple providers are available, calls walk the priority chain: configured default → VISION_MCP_PROVIDER_PRIORITY list → table order (unavailable providers are skipped).

• each provider is retried up to VISION_MCP_MAX_RETRIES times on provider errors (upstream failures, timeouts)

• after a provider exhausts its retries, the next available provider in the chain is tried, up to VISION_MCP_MAX_FALLBACKS fallbacks

• only provider errors trigger retry/fallback; config or image errors fail fast

• an explicitly requested provider argument is tried alone (no fallback)

• when everything fails, the error lists every provider attempted and its last error

Also available as argv: --provider-priority=..., --max-retries=N, --max-fallbacks=N (beat env vars).

MCP startup arguments (argv)

Every provider's apiKey / baseUrl / model can be overridden via startup arguments (higher priority than environment variables), format --<provider>-<field>:

node dist/index.js \
  --openai-api-key=sk-xxx \
  --openai-base-url=https://my-gateway.example.com/v1 \
  --openai-model=gpt-4o-mini \
  --dashscope-api-key=sk-xxx \
  --default-provider=dashscope

• Global: --default-provider <name> / --default-model <name>

• Per provider: --<provider>-api-key, --<provider>-base-url, --<provider>-model (equals or space form both work)

• Any OpenAI-compatible third-party service: wire it up in one line with --openai-compat-base-url + --openai-compat-api-key + --openai-compat-model; or point any built-in provider's base-url at a mirror/proxy

Priority: tool args provider/model > startup args (per-provider > global default) > environment variables > provider built-in defaults.

Tools

image accepts: local path / http(s):// URL / data: URI / raw base64, auto-sniffed.

Security note: URL downloads are SSRF-protected — every hop (including redirects) is validated and URLs resolving to loopback, private, or link-local addresses are blocked (hint in the error explains why).

Provider integration (three-tier fallback chain)

Adding a provider: for OpenAI-compatible endpoints, add one row to RULES in src/core/config.ts plus one mapping in the factory table in src/index.ts — zero new code. Official SDK or native fetch implementations: see SPEC §1.

Development

pnpm check        # biome checks
pnpm test         # rstest unit tests (injected mocks, no network)
pnpm build        # rslib build

Real-call smoke tests (only run against providers whose keys are configured; skipped otherwise):

OPENAI_API_KEY=sk-... pnpm exec rstest tests/e2e

Architecture

src/
├── index.ts            # Entry: composition root, stdio startup
├── core/               # Abstraction: interfaces / image loading / config / registry
├── providers/          # Adapters: official SDK or compatible endpoints, protocol conversion only
└── server/tools.ts     # MCP tool layer: zod validation + error mapping

Full spec: SPEC.md.