cv-mcp

Minimal MCP server focused on computer vision: image recognition and metadata generation via OpenRouter (Gemini 2.5 family).

Goals

Keep it tiny and composable
Single tool: caption an image via URL or local file
No DB or app logic

Structure

src/cv_mcp/captioning/openrouter_client.py – image analysis client
src/cv_mcp/metadata/ – prompts, JSON schema, and pipeline runner
src/cv_mcp/mcp_server.py – MCP server exposing tools
cli/caption_image.py – optional CLI to test captioning locally

Env vars

OPENROUTER_API_KEY

Dotenv

Put OPENROUTER_API_KEY in a local .env file (see .env.example).
CLI scripts and the MCP server auto-load .env if present.

Install

pip install -e . (or pip install .)

⚠️ Development Note: If you have the package installed via pip install, uninstall it before working with the local development version to avoid import conflicts. Use pip uninstall cv-mcp first, then run commands directly from the repo directory.

Run MCP server (stdio)

Console script: cv-mcp-server (provides an MCP stdio server)
Configure your MCP client to launch cv-mcp-server.

MCP integration (Claude Desktop)

Add to Claude Desktop config (see their docs for the config location): { "mcpServers": { "cv-mcp": { "command": "cv-mcp-server", "env": { "OPENROUTER_API_KEY": "sk-or-..." } } } }
After saving, restart Claude Desktop and enable the tool.

Tools

caption_image: one-off caption (kept for compatibility)
alt_text: short alt text (<= 20 words)
dense_caption: detailed 2–6 sentence caption
image_metadata: structured JSON metadata with alt + caption. Params:
- mode: double (default) uses 2 calls: vision (alt+caption) + text-only (metadata). triple uses vision for both steps.
- caption_override: supply your own dense caption; skips the vision caption step.

MCP tool reference

Server: cv-mcp (stdio)
caption_image(image_url|file_path, prompt?, backend?, local_model_id?) -> string
alt_text(image_url|file_path, max_words?) -> string
dense_caption(image_url|file_path) -> string
image_metadata(image_url|file_path, caption_override?, config_path?) -> { alt_text, caption, metadata }

Examples

MCP call (OpenRouter): {"image_url": "https://example.com/image.jpg"}
MCP call (local): {"backend": "local", "file_path": "./image.jpg"}

Quick test (CLI)

URL: python cli/caption_image.py --image-url https://example.com/img.jpg
File: python cli/caption_image.py --file-path ./image.png

Metadata pipeline (CLI)

Double (default):
- python cli/image_metadata.py --image-url https://example.com/img.jpg --mode double
- Local alt+caption (still requires OpenRouter for metadata):
  - python cli/image_metadata.py --image-url https://example.com/img.jpg --mode double --ac-backend local
Triple (vision metadata):
- python cli/image_metadata.py --image-url https://example.com/img.jpg --mode triple
- Fully local (no OpenRouter required):
  - python cli/image_metadata.py --image-url https://example.com/img.jpg --mode triple --ac-backend local --meta-vision-backend local
With existing caption (skips the caption step):
- python cli/image_metadata.py --image-url https://example.com/img.jpg --caption-override "<dense caption>" --mode double
Custom model config (JSON with caption_model, metadata_text_model, metadata_vision_model):
- python cli/image_metadata.py --image-url https://example.com/img.jpg --config-path ./my_models.json --mode double

Schema & vocab

JSON schema (lean): src/cv_mcp/metadata/schema.json
Controlled vocab (non-binding reference): src/cv_mcp/metadata/vocab.json

Global config

Root file: cv_mcp.config.json (auto-detected from project root / CWD)
Env override: set CV_MCP_CONFIG=/path/to/config.json
Keys (renamed for clarity):
- caption_model: vision model for alt+caption (OpenRouter)
- metadata_text_model: text model for metadata (double mode)
- metadata_vision_model: vision model for metadata (triple mode)
- caption_backend: openrouter (default) or local for alt/dense/AC steps
- metadata_vision_backend: openrouter (default) or local for triple mode
- local_vlm_id: default local VLM (e.g. Qwen/Qwen2.5-VL-7B-Instruct)
- Backwards-compat: legacy keys (ac_model, meta_text_model, meta_vision_model, ac_backend, meta_vision_backend, local_model_id) are still accepted.
Packaged defaults still live at src/cv_mcp/metadata/config.json and are used if no root config is found.
You can still provide a custom config file per-call via --config-path or the config_path tool param.

Local backends (optional)

Install optional deps: pip install .[local]
Global default: set "caption_backend": "local" (and optionally "metadata_vision_backend": "local") in cv_mcp.config.json
Use with MCP: pass backend: "local" in the tool params (overrides global)
Use with CLI: add --backend local and optionally --local-model-id Qwen/Qwen2-VL-2B-Instruct (overrides global)
Requires a locally available model (default: Qwen/Qwen2-VL-2B-Instruct via HF cache)
Or run without transformers using Ollama (no Python ML deps):
- Install and run Ollama; pull a vision model (e.g., ollama pull qwen2.5-vl)
- Use backend ollama and set models in the config (e.g., caption_model: "qwen2.5-vl")
- CLI example (triple, fully local):
  - python cli/image_metadata.py --image-url https://... --mode triple --caption-backend ollama --metadata-vision-backend ollama --config-path ./configs/triple_ollama_qwen.json
- Configure host with --ollama-host http://localhost:11434 if not default

Per-call overrides (CLI)

Metadata CLI now supports per-call backend overrides without editing global config:
- --caption-backend local|openrouter|ollama (legacy: --ac-backend)
- --metadata-vision-backend local|openrouter|ollama (legacy: --meta-vision-backend)
- --local-vlm-id Qwen/Qwen2.5-VL-7B-Instruct (legacy: --local-model-id)
- --ollama-host http://localhost:11434

Justfile tasks

A Justfile provides quick test scenarios. Use URL-only inputs, e.g. just double_flash https://example.com/img.jpg.
Scenarios included:
- double_flash: Gemini 2.5 Flash for both steps
- double_pro: Gemini 2.5 Pro for both steps
- double_mixed_pro_text: Flash for vision alt+caption, Pro for text metadata (recommended mix for JSON reliability)
- triple_flash / triple_pro: Flash/Pro for both vision steps
- double_qwen_local <url> <qwen_id>: Local Qwen 2.5 VL for vision step, Pro for text metadata
- triple_qwen_local <url> <qwen_id>: Fully local Qwen 2.5 VL for both vision steps
- Convenience (no extra args):
  - double_qwen2b_local <url> / triple_qwen2b_local <url>
  - double_qwen7b_local <url> / triple_qwen7b_local <url>

Recommendation for mixed double

Put Gemini 2.5 Pro on the text metadata step and Flash on the vision alt+caption step. The metadata step benefits from better structured-JSON compliance and reasoning, while Flash keeps latency/cost down for the vision caption.
OpenRouter key requirements:
- Double mode always requires OPENROUTER_API_KEY (text LLM for metadata).
- Triple mode requires OPENROUTER_API_KEY unless both --ac-backend local and --meta-vision-backend local are set.

Examples

MCP tool (local): {"backend": "local", "file_path": "./image.jpg"}
CLI (local): python cli/caption_image.py --file-path ./image.jpg --backend local

Troubleshooting

401/403 from OpenRouter: ensure OPENROUTER_API_KEY is set and valid.
Model selection: prefer cv_mcp.config.json at project root; or pass --config-path.
Large images: remote images are downloaded and sent as base64; ensure the URL is accessible.
Local backend: install optional deps pip install .[local] and ensure model is present/cached.

Changelog