Vision MCP Server

English | 简体中文

lm-visual-mcp gives text-only LLMs and coding agents visual input through the Model Context Protocol. It is built from four modules — mcp, server, providers, and vision — and also ships a transparent HTTP proxy that rewrites image blocks in OpenAI or Anthropic requests into text descriptions before forwarding them upstream, plus Claude Code Auto classifier interoperability.

This version (v0.2.0) supports image recognition only. Video input is no longer declared or accepted anywhere in the stack.

Modules

Hooks

A hook's basic interface is process(ctx) -> HookResult: it may rewrite the request and let it continue down the pipeline, or intercept it by returning a response that goes straight back to the client. Hooks may also implement process_response to rewrite the upstream response (used by the classifier hook).

• Image hook — detects image-bearing requests, describes each image once (SHA-256 cache) through the image chain, and replaces the image block with text. Every rewritten block records the image's absolute local path ([Image N: /abs/path.png]), and staged files persist, so the text model can reference or re-submit the image later.

• Classifier hook — detects Claude Code Auto classifier requests and delegates to the classifier chain. Only API providers that implement classifier handling (rewrite_classifier_request / rewrite_classifier_response) rewrite these; local CLI providers (agy, codex) pass them through byte-for-byte untouched.

Both hooks accept a models allowlist — empty = apply to all models, non-empty = only the listed models run through the router; everything else passes through untouched.

Providers, dual chains & fallback

The top-level providers: section defines provider instances (the single source of truth, referenced by name). vision then references those names in two independent execution chains:

• image_chain — image analysis fallback order (first success wins).

• classifier_chain — classifier handling order (first provider that reports a changed rewrite wins; if none implements classifier handling, requests pass through untouched).

The router never hardcodes providers — type is resolved through a registry, so adding one is a class + one registry line + config.

Rate limiting lives inside each provider (rate_limit: {rpm, concurrency} per entry, both optional). When a limit is hit the provider raises rate_limited and the router immediately downgrades to the next provider in its chain.

• agy — AGY CLI (-p + --add-dir + sandbox), unsupported-vision verdict caching. IMAGE only; no classifier handling.

• codex — codex exec with --output-schema, read-only sandbox. IMAGE only.

• gemini — google-genai API (api_key_env: GEMINI_API_KEY). IMAGE + classifier (honors disable_thinking).

• opencode — direct OpenAI-compatible API; mode: go (default, https://opencode.ai/zen/go/v1) or mode: zen, base_url overrides mode. IMAGE + classifier; no local CLI required.

• volcengine — Volcano Ark; mode: agent (Anthropic Messages /v1/messages over api/plan), mode: coding (api/coding), or mode: api (OpenAI chat-completions api/v3). IMAGE + classifier.

Related MCP server: Vision MCP Server

Architecture

MCP client process (agent config: --start-server / --no-start-server)
    │ stdio
    ▼
mcp module (thin client)
    │ loopback HTTP  POST /vision/analyze
    ▼
server module (shared singleton)
    ├── vision endpoint ──► vision module
    │                          ├── concurrency gate
    │                          └── router walks image_chain: provider₁ → provider₂ → …
    │                                (each with its own rpm/concurrency limiter;
    │                                 limit hit → fall back to the next)
    └── hook proxy  /proxy/<proto>/<base64url>[/suffix]
           ├── image hook      → image chain (description rewrite, model-allowlist)
           ├── classifier hook → classifier chain (API-provider rewrite, model-allowlist /
           │                      byte-level passthrough when no provider handles it)
           └── byte-level passthrough when no hook applies

Quick start

# agent MCP config (stdio) — starts the shared server if absent:
lm-visual-mcp

# …or never start the server from the MCP process (use an already-running one):
lm-visual-mcp --no-start-server      # env: LM_VISUAL_MCP_START_SERVER=0

lm-visual-mcp start | stop | restart  # manage the server singleton
lm-visual-mcp server                  # run the server in the foreground
lm-visual-mcp doctor                  # inspect configuration and providers

Copy config.example.yaml to lm-visual-mcp.yaml (or ~/.config/lm-visual-mcp/) to configure the listen address, hooks, providers and the two chains. Root nodes are server / hooks / providers / vision / media / logging. There is no mcp: section in the file — the start-server decision belongs to the agent's MCP config, not to the YAML.

Example (top-level providers defines instances; vision declares the chains):

server:
  host: 127.0.0.1
  port: 8787

hooks:
  image:      { enabled: true, models: [] }   # models empty = all models
  classifier: { enabled: true, models: [] }

providers:
  - name: agy
    type: agy
    command: agy
    model: gemini-3.6-flash
    effort: high
    rate_limit: { rpm: 30, concurrency: 2 }
  - name: gemini
    type: gemini
    api_key_env: GEMINI_API_KEY
    disable_thinking: true
  - name: opencode
    type: opencode
    mode: go                              # go | zen
    api_key_env: OPENCODE_API_KEY
  - name: volcengine
    type: volcengine
    mode: agent                           # agent | coding | api
    api_key_env: VOLCENGINE_API_KEY

vision:
  timeout: 120
  max_concurrency: 2
  image_chain: [agy, gemini, opencode]      # first success wins
  classifier_chain: [gemini]                # only API providers belong here

Local CLI providers (agy, codex) have no classifier handling; put only API providers (gemini / opencode / volcengine) on the classifier_chain.

Tools

ui_to_artifact, extract_text_from_screenshot, diagnose_error_screenshot, understand_technical_diagram, analyze_data_visualization, ui_diff_check, analyze_image (+ image_analysis alias). Provider, model, credentials, fallback policy, timeout are server configuration and never appear in tool schemas.