vision-bridge-mcp

Vision sidecar MCP server — gives text-only LLMs the ability to see images. Supports OpenAI AND Anthropic API formats natively. Includes model-capability routing skill.

Why?

Most LLMs are text-only — they cannot see images. This MCP server bridges that gap by forwarding images to a vision-capable model and returning text results. It works with any OpenAI-compatible or Anthropic-compatible API endpoint.

When paired with the vision-sidecar skill, it automatically routes based on the host model's capabilities:

Exception: when the system clipboard has an image and the conversation has no path/URL/attachment, even multimodal host models may pass image="clipboard".

Features

• ✅ Three tools: analyze_image, ocr_image, compare_images

• ✅ Dual protocol: OpenAI chat/completions AND Anthropic messages format

• ✅ Clipboard support: Windows (PowerShell) + macOS (Swift)

• ✅ SHA256 file cache with configurable TTL

• ✅ URL download retry: auto-downloads remote URLs to base64 when passthrough fails

• ✅ Reasoning model fallback: extracts reasoning_content when content is null

• ✅ Full-chain timeout: connection + headers + body reading

• ✅ Safety limits: 16MB response / 20MB image / 1MB error detail

• ✅ Typed errors: VisionInputError / VisionApiError / VisionTimeoutError

• ✅ Comprehensive tests: 30+ unit tests + end-to-end smoke tests

• ✅ Minimal dependencies: only @modelcontextprotocol/sdk + zod (declared in package.json)

Quick Start

Option A: npx (npm package)

1. Ensure node ≥ 18 is in your PATH.

2. Set environment variables:

export VISION_API_BASE_URL=https://api.example.com/v1   # OpenAI: ends with /v1; Anthropic: base without /v1
export VISION_API_KEY=sk-...                             # API key
export VISION_MODEL=gpt-4o                               # Vision model name
# Optional: export VISION_API_FORMAT=anthropic            # openai (default) or anthropic

3. Register in your MCP client config:

{
  "id": "vision-bridge-mcp",
  "transport": "stdio",
  "command": "npx",
  "args": ["vision-bridge-sidecar"],
  "env": {
    "VISION_API_BASE_URL": "https://api.example.com/v1",
    "VISION_API_KEY": "your-key",
    "VISION_MODEL": "gpt-4o"
  },
  "enabled": true
}

Option B: from source

git clone https://github.com/Catapult291/vision-bridge-mcp.git
cd vision-bridge-mcp
npm install

Register with command: "node", args: ["server.js"], cwd: "/path/to/vision-bridge-mcp".

Configuration

Startup validates the first three variables; missing ones cause a readable error and exit (code 1).

Tools

analyze_image

Prerequisite: Only call when the host model lacks multimodal vision. If the host model is multimodal, use its native image understanding.

• image (required, string): Local file path / http(s) URL / base64 dataURL / clipboard.

  • Local path: infers MIME from extension (png/jpg/jpeg/gif/webp/bmp), converts to base64 dataURL.

  • http(s) URL: passed as image_url directly.

  • dataURL: only image/* base64 encoding accepted.

  • clipboard / clip / pasteboard: reads current system clipboard image (Windows: scripts/clipboard.ps1, macOS: scripts/clipboard.swift), writes to temp PNG, then normalizes. Linux not supported.

• prompt (optional, string): Custom recognition instruction. Default: "Describe this image in detail."

• Returns: success { content: [{ type: "text", text }] }; failure { content: [{ type: "text", text: "[vision_error] ..." }], isError: true }.

Internal request (split by VISION_API_FORMAT):

• OpenAI: POST {base}/chat/completions, image as image_url part, auth Authorization: Bearer.

• Anthropic: POST {base}/v1/messages, image as image block (source: {type: base64, media_type, data} or {type: url, url}), auth x-api-key + anthropic-version: 2023-06-01 (also sends Authorization: Bearer for compatibility).

Default timeout: 60s (covers connection + body reading).

Safety limits: API response 16MB, image download 20MB (content-length precheck + actual size recheck).

Behavior notes (from real model testing):

• Reasoning models may return content: null with answer in reasoning_content — automatically falls back.

• http(s) URL passthrough failure with media/download error → auto-downloads to base64 and retries once.

ocr_image

• image (required, string): Same normalization as analyze_image.

• languages (optional, string): Language hints (e.g. zh,en).

• format (optional, enum): plain (default, plain text preserving layout) / markdown (preserves headings/lists/tables) / json (returns blocks array with text + type).

• Uses image_url.detail = "high" internally; prompt injected per format.

compare_images

• images (required, array, 2–4): Each supports local path / http(s) URL / dataURL / clipboard.

• prompt (optional, string): Custom comparison instruction. Default: "Compare these images and describe their differences and similarities."

• Single user message with text + multiple image_url parts (detail = "auto").

• If any URL fails with media/download error, all URLs are downloaded to base64 and retried once.

Vision Sidecar Skill

The vision-sidecar skill provides model-capability routing. When enabled in your MCP client:

• Host model has multimodal → uses native image understanding (no MCP call)

• Host model is text-only → calls this MCP's analyze_image

• Exception: clipboard reading available for multimodal host models

Without the skill, the host model's behavior is completely unchanged — zero intrusion.

See skill/vision-sidecar.md for the skill file.

Caching

Enabled by default. Caches vision API results for identical "image + prompt" combinations.

• Key: SHA256(image identifier + "::" + prompt). Local files/dataURLs hashed by base64 content; http(s) URLs hashed by URL string.

• Storage: One JSON file per key ({ result, cachedAt }), stored in cache directory.

• TTL: Default 1 hour. Expired entries auto-deleted on next access.

• Disable: VISION_CACHE_TTL=0 (or negative).

• Note: Key does not include model name. After switching VISION_MODEL, TTL-period may return cached results from the old model — clear cache directory when switching models.

Testing

cd vision-bridge-mcp
node --test

• test/vision.test.js: Core library unit tests (input normalization / message body / API calls / error mapping / timeout / cache / URL retry / OCR / clipboard).

• test/cache.test.js: Cache module tests (key stability / hit / expiry / corrupted JSON / backward compat).

• test/smoke.test.mjs: End-to-end smoke test — spawns real server.js via stdio, uses local HTTP stub to simulate vision model, validates tools/list and tool calls.

Comparison with Other Vision MCPs

See docs/COMPARISON.md for a detailed comparison with other vision MCP projects.

For a deep dive into the internals — data flow, input normalization, dual-protocol message construction, URL retry, and caching — see docs/ARCHITECTURE.md.

License

MIT