# Vision MCP Server

A Model Context Protocol (MCP) server that gives **vision understanding to agents connected to non-multimodal models** (DeepSeek, older GPT-4, local small models, etc.): the agent hands an image to the MCP tool, the server calls a vision model, and returns text.

Supports major providers in China and the US plus any OpenAI-compatible endpoint. Official SDKs first, abstraction before implementation, zero-intrusion provider additions.

> 中文文档见 [README.zh-CN.md](./README.zh-CN.md)

## Features

- **4 tools**: `analyze_image` / `describe_image` / `ocr_image` / `list_providers`, all returning plain Markdown text
- **13 built-in providers**: OpenAI / Anthropic / Google Gemini / Qwen (DashScope) / Zhipu / Doubao (Volcengine) / ERNIE (Qianfan) / StepFun / Ollama / Alibaba Bailian / SiliconFlow / OpenRouter / custom OpenAI-compatible endpoint
- **Up to 9 images per call** (configurable via `VISION_MCP_MAX_IMAGES`): local path / http(s) URL / base64 (data URI or raw base64), auto-sniffed, types mixable
- **Three-tier fallback chain**: official SDK → OpenAI-compatible endpoint → native fetch (see SPEC §1)
- **Stateless**: every call is independent; images and results are never cached; keys are read from environment variables only

## Quick start

**Option A: npx (published to npm, no repo needed)**

```bash
npx -y @inferai/vision-mcp
```

**Option B: local build**

```bash
git clone <repo> && cd vision-mcp
pnpm install
pnpm build
node dist/index.js
```

### MCP configuration examples (stdio)

The server speaks the **stdio transport**: the MCP client spawns the process and exchanges JSON-RPC messages over stdin/stdout. Configure it wherever your client defines MCP servers:

- **Claude Code**: project-level `.mcp.json` or user-level `~/.claude.json` (`mcpServers` key)
- **Claude Desktop**: `claude_desktop_config.json`
- **Any MCP client** (Cursor, self-built agents, etc.): same structure

#### Windows

On Windows `npx` resolves to `npx.cmd`, and MCP clients that spawn processes without a shell can't run it directly — wrap it in `cmd /c`:

```json
{
  "mcpServers": {
    "vision-mcp": {
      "command": "cmd",
      "args": ["/c", "npx", "-y", "@inferai/vision-mcp"],
      "env": {
        "OPENAI_API_KEY": "sk-...",
        "DASHSCOPE_API_KEY": "sk-..."
      }
    }
  }
}
```

Local development (adjust the path; `--env-file-if-exists=.env` loads `.env` natively):

```json
{
  "mcpServers": {
    "vision-mcp": {
      "command": "node",
      "args": [
        "--env-file-if-exists=.env",
        "C:\\path\\to\\vision-mcp\\dist\\index.js"
      ],
      "env": {
        "OPENAI_API_KEY": "sk-..."
      }
    }
  }
}
```

#### Linux / macOS

`npx` runs directly:

```json
{
  "mcpServers": {
    "vision-mcp": {
      "command": "npx",
      "args": ["-y", "@inferai/vision-mcp"],
      "env": {
        "OPENAI_API_KEY": "sk-...",
        "DASHSCOPE_API_KEY": "sk-..."
      }
    }
  }
}
```

Local development (adjust the path; `--env-file-if-exists=.env` loads `.env` natively):

```json
{
  "mcpServers": {
    "vision-mcp": {
      "command": "node",
      "args": [
        "--env-file-if-exists=.env",
        "/absolute/path/to/vision-mcp/dist/index.js"
      ],
      "env": {
        "OPENAI_API_KEY": "sk-..."
      }
    }
  }
}
```

With startup arguments (override provider defaults via argv, see below; on Windows prefix the `command`/`args` with `cmd /c`):

```json
{
  "mcpServers": {
    "vision-mcp": {
      "command": "npx",
      "args": [
        "-y",
        "@inferai/vision-mcp",
        "--default-provider=dashscope",
        "--siliconflow-api-key=sk-...",
        "--siliconflow-model=Qwen/Qwen2.5-VL-7B-Instruct"
      ],
      "env": {
        "DASHSCOPE_API_KEY": "sk-..."
      }
    }
  }
}
```

**stdio notes:**
- stdout carries the MCP protocol only — the server never prints logs there; diagnostics go to stderr
- the client manages the process lifecycle (spawn on start, kill on exit); no daemon needed
- first `npx` run downloads the package and may take a few seconds
- env variables can also come from the shell environment if the client inherits it (no `env` block needed)

Debug with MCP Inspector:

```bash
pnpm dlx @modelcontextprotocol/inspector node dist/index.js --xxx-api-key=xxx --xxx2-api-key=xxx
```

### Setting variables

1. **MCP config `env` block** (recommended, most reliable across platforms) — write the variables into the `env` object above
2. **`.env` file** (local development) — copy [`.env.example`](./.env.example) to `.env`, fill it in, then `node --env-file-if-exists=.env dist/index.js` (Node 22 native, no dotenv needed)
3. **Shell export** — `export OPENAI_API_KEY=sk-xxx` then run

Providers without keys show as unavailable in `list_providers` and report the missing variable when called.

### Publishing (before npx works)

```bash
pnpm publish          # or pnpm release (changeset flow)
```

## Environment variables

Every provider's `API_KEY`, `BASE_URL`, and `MODEL` support environment overrides (convention: `<PROVIDER_PREFIX>_API_KEY` / `<PROVIDER_PREFIX>_BASE_URL` / `<PROVIDER_PREFIX>_MODEL`):

| Provider | Environment variables | Default model |
|---|---|---|
| OpenAI | `OPENAI_API_KEY`, `OPENAI_BASE_URL`, `OPENAI_MODEL` | `gpt-4o` |
| Anthropic | `ANTHROPIC_API_KEY`, `ANTHROPIC_BASE_URL`, `ANTHROPIC_MODEL` | `claude-sonnet-4-5` |
| Google Gemini | `GEMINI_API_KEY`, `GEMINI_BASE_URL`, `GEMINI_MODEL` | `gemini-2.5-flash` |
| Alibaba DashScope | `DASHSCOPE_API_KEY`, `DASHSCOPE_BASE_URL`, `DASHSCOPE_MODEL` | `qwen-vl-max` |
| Zhipu | `ZHIPU_API_KEY`, `ZHIPU_BASE_URL`, `ZHIPU_MODEL` | `glm-4v-flash` (free) |
| Volcengine Doubao | `VOLCENGINE_ARK_API_KEY`, `VOLCENGINE_ARK_BASE_URL`, `VOLCENGINE_ARK_MODEL` | `doubao-1.5-vision-pro` |
| Baidu Qianfan | `QIANFAN_API_KEY`, `QIANFAN_SECRET_KEY`, `QIANFAN_BASE_URL`, `QIANFAN_MODEL` | `ernie-4.5-vl-8k` |
| StepFun | `STEPFUN_API_KEY`, `STEPFUN_BASE_URL`, `STEPFUN_MODEL` | `step-1v` |
| Ollama (local) | `OLLAMA_BASE_URL`*, `OLLAMA_MODEL`* | — (no built-in default; endpoint and model must be set) |
| Alibaba Bailian | `BAILIAN_API_KEY`, `BAILIAN_BASE_URL` (default DashScope compatible mode), `BAILIAN_MODEL` | `qwen-vl-max` |
| SiliconFlow | `SILICONFLOW_API_KEY`, `SILICONFLOW_BASE_URL` (default `https://api.siliconflow.cn/v1`), `SILICONFLOW_MODEL` | `Qwen/Qwen2.5-VL-72B-Instruct` |
| OpenRouter | `OPENROUTER_API_KEY`, `OPENROUTER_BASE_URL` (default `https://openrouter.ai/api/v1`), `OPENROUTER_MODEL` | `openai/gpt-4o` |
| Custom compatible | `OPENAI_COMPAT_BASE_URL`*, `OPENAI_COMPAT_API_KEY`?, `OPENAI_COMPAT_MODEL`* | — |

> `?` = optional (has a built-in default); `*` = required.

Global configuration:

| Environment variable | Default | Description |
|---|---|---|
| `VISION_MCP_DEFAULT_PROVIDER` | first available | Default provider |
| `VISION_MCP_DEFAULT_MODEL` | provider default | Default model |
| `VISION_MCP_PROVIDER_PRIORITY` | table order | Provider priority (comma-separated, high first, e.g. `openai,dashscope,zhipu`) |
| `VISION_MCP_MAX_RETRIES` | 0 (off) | Per-provider retry count before falling back |
| `VISION_MCP_MAX_FALLBACKS` | 0 (off) | Max provider fallbacks before giving up |
| `VISION_MCP_MAX_IMAGE_BYTES` | 20 MB | Image size limit |
| `VISION_MCP_MAX_IMAGES` | 9 | Max images per tool call |
| `VISION_MCP_TIMEOUT_MS` | 60000 | Download & request timeout (ms) |

### Fallback chain

When multiple providers are available, calls walk the priority chain: configured default → `VISION_MCP_PROVIDER_PRIORITY` list → table order (unavailable providers are skipped).

- each provider is retried up to `VISION_MCP_MAX_RETRIES` times on provider errors (upstream failures, timeouts)
- after a provider exhausts its retries, the next available provider in the chain is tried, up to `VISION_MCP_MAX_FALLBACKS` fallbacks
- only provider errors trigger retry/fallback; config or image errors fail fast
- an explicitly requested `provider` argument is tried alone (no fallback)
- when everything fails, the error lists every provider attempted and its last error

Also available as argv: `--provider-priority=...`, `--max-retries=N`, `--max-fallbacks=N` (beat env vars).

### MCP startup arguments (argv)

Every provider's `apiKey` / `baseUrl` / `model` can be overridden via startup arguments (higher priority than environment variables), format `--<provider>-<field>`:

```bash
node dist/index.js \
  --openai-api-key=sk-xxx \
  --openai-base-url=https://my-gateway.example.com/v1 \
  --openai-model=gpt-4o-mini \
  --dashscope-api-key=sk-xxx \
  --default-provider=dashscope
```

- Global: `--default-provider <name>` / `--default-model <name>`
- Per provider: `--<provider>-api-key`, `--<provider>-base-url`, `--<provider>-model` (equals or space form both work)
- Any OpenAI-compatible third-party service: wire it up in one line with `--openai-compat-base-url` + `--openai-compat-api-key` + `--openai-compat-model`; or point any built-in provider's `base-url` at a mirror/proxy

Priority: tool args `provider`/`model` > startup args (per-provider > global default) > environment variables > provider built-in defaults.

## Tools

| Tool | Arguments | Description |
|---|---|---|
| `analyze_image` | `images`*, `prompt`?, `provider`?, `model`? | General image analysis |
| `describe_image` | `images`*, `provider`?, `model`? | Describe image content (default instruction) |
| `ocr_image` | `images`*, `language`? (`auto`/`zh`/`en`/`zh-en`), `provider`?, `model`? | OCR, preserving layout |
| `list_providers` | — | Provider list and configuration status |

`images` accepts a single image or an array (up to `VISION_MCP_MAX_IMAGES`, default 9): local path / `http(s)://` URL / `data:` URI / raw base64, auto-sniffed. Multiple images are seen by the model in the given order (compare, diff, or combine them).

**Security note:** URL downloads are SSRF-protected — every hop (including redirects) is validated and URLs resolving to loopback, private, or link-local addresses are blocked (hint in the error explains why).

## Provider integration (three-tier fallback chain)

| provider | Integration | Notes |
|---|---|---|
| `openai` / `stepfun` / `ollama` / `bailian` / `siliconflow` / `openrouter` / `openai-compat` | **OpenAI-compatible adapter** (openai SDK) | One adapter, configurable baseURL |
| `anthropic` | **Official SDK** @anthropic-ai/sdk | messages + image content block |
| `gemini` | **Official SDK** @google/generative-ai | generateContent + inlineData |
| `dashscope` | **Native fetch** | official npm package has no vision; direct multimodal-generation API |
| `zhipu` | **Native fetch** | official SDK accepts string content only; direct v4 API |
| `volcengine` | **Native fetch** | official openapi is a management plane; direct Ark API |
| `qianfan` | **Native fetch** | official SDK is string-only; AK/SK → token → v2 API |

Adding a provider: for OpenAI-compatible endpoints, add one row to `RULES` in `src/core/config.ts` plus one mapping in the factory table in `src/index.ts` — zero new code. Official SDK or native fetch implementations: see SPEC §1.

## Development

```bash
pnpm check        # biome checks
pnpm test         # rstest unit tests (injected mocks, no network)
pnpm build        # rslib build
```

Real-call smoke tests (only run against providers whose keys are configured; skipped otherwise):

```bash
OPENAI_API_KEY=sk-... pnpm exec rstest tests/e2e
```

## Architecture

```
src/
├── index.ts            # Entry: composition root, stdio startup
├── core/               # Abstraction: interfaces / image loading / config / registry
├── providers/          # Adapters: official SDK or compatible endpoints, protocol conversion only
└── server/tools.ts     # MCP tool layer: zod validation + error mapping
```

Full spec: [SPEC.md](./SPEC.md).