cv-mcp

Minimal MCP server focused on computer vision: image recognition and metadata generation via OpenRouter (Gemini 2.5 family).

Goals

• Keep it tiny and composable

• Single tool: caption an image via URL or local file

• No DB or app logic

Structure

• src/cv_mcp/captioning/openrouter_client.py – image analysis client

• src/cv_mcp/metadata/ – prompts, JSON schema, and pipeline runner

• src/cv_mcp/mcp_server.py – MCP server exposing tools

• cli/caption_image.py – optional CLI to test captioning locally

Env vars

• OPENROUTER_API_KEY

Dotenv

• Put OPENROUTER_API_KEY in a local .env file (see .env.example).

• CLI scripts and the MCP server auto-load .env if present.

Install

• pip install -e . (or pip install .)

⚠️ Development Note: If you have the package installed via pip install, uninstall it before working with the local development version to avoid import conflicts. Use pip uninstall cv-mcp first, then run commands directly from the repo directory.

Run MCP server (stdio)

• Console script: cv-mcp-server (provides an MCP stdio server)

• Configure your MCP client to launch cv-mcp-server.

MCP integration (Claude Desktop)

• Add to Claude Desktop config (see their docs for the config location): { "mcpServers": { "cv-mcp": { "command": "cv-mcp-server", "env": { "OPENROUTER_API_KEY": "sk-or-..." } } } }

• After saving, restart Claude Desktop and enable the tool.

Tools

• caption_image: one-off caption (kept for compatibility)

• alt_text: short alt text (<= 20 words)

• dense_caption: detailed 2–6 sentence caption

• image_metadata: structured JSON metadata with alt + caption. Params:

  • mode: double (default) uses 2 calls: vision (alt+caption) + text-only (metadata). triple uses vision for both steps.

  • caption_override: supply your own dense caption; skips the vision caption step.

MCP tool reference

• Server: cv-mcp (stdio)

• caption_image(image_url|file_path, prompt?, backend?, local_model_id?) -> string

• alt_text(image_url|file_path, max_words?) -> string

• dense_caption(image_url|file_path) -> string

• image_metadata(image_url|file_path, caption_override?, config_path?) -> { alt_text, caption, metadata }

Examples

• MCP call (OpenRouter): {"image_url": "https://example.com/image.jpg"}

• MCP call (local): {"backend": "local", "file_path": "./image.jpg"}

Quick test (CLI)

• URL: python cli/caption_image.py --image-url https://example.com/img.jpg

• File: python cli/caption_image.py --file-path ./image.png

Metadata pipeline (CLI)

• Double (default):

  • python cli/image_metadata.py --image-url https://example.com/img.jpg --mode double

  • Local alt+caption (still requires OpenRouter for metadata):

    • python cli/image_metadata.py --image-url https://example.com/img.jpg --mode double --ac-backend local

• Triple (vision metadata):

  • python cli/image_metadata.py --image-url https://example.com/img.jpg --mode triple

  • Fully local (no OpenRouter required):

    • python cli/image_metadata.py --image-url https://example.com/img.jpg --mode triple --ac-backend local --meta-vision-backend local

• With existing caption (skips the caption step):

  • python cli/image_metadata.py --image-url https://example.com/img.jpg --caption-override "<dense caption>" --mode double

• Custom model config (JSON with caption_model, metadata_text_model, metadata_vision_model):

  • python cli/image_metadata.py --image-url https://example.com/img.jpg --config-path ./my_models.json --mode double

Schema & vocab

• JSON schema (lean): src/cv_mcp/metadata/schema.json

• Controlled vocab (non-binding reference): src/cv_mcp/metadata/vocab.json

Global config

• Root file: cv_mcp.config.json (auto-detected from project root / CWD)

• Env override: set CV_MCP_CONFIG=/path/to/config.json

• Keys (renamed for clarity):

  • caption_model: vision model for alt+caption (OpenRouter)

  • metadata_text_model: text model for metadata (double mode)

  • metadata_vision_model: vision model for metadata (triple mode)

  • caption_backend: openrouter (default) or local for alt/dense/AC steps

  • metadata_vision_backend: openrouter (default) or local for triple mode

  • local_vlm_id: default local VLM (e.g. Qwen/Qwen2.5-VL-7B-Instruct)

  • Backwards-compat: legacy keys (ac_model, meta_text_model, meta_vision_model, ac_backend, meta_vision_backend, local_model_id) are still accepted.

• Packaged defaults still live at src/cv_mcp/metadata/config.json and are used if no root config is found.

• You can still provide a custom config file per-call via --config-path or the config_path tool param.

Local backends (optional)

• Install optional deps: pip install .[local]

• Global default: set "caption_backend": "local" (and optionally "metadata_vision_backend": "local") in cv_mcp.config.json

• Use with MCP: pass backend: "local" in the tool params (overrides global)

• Use with CLI: add --backend local and optionally --local-model-id Qwen/Qwen2-VL-2B-Instruct (overrides global)

• Requires a locally available model (default: Qwen/Qwen2-VL-2B-Instruct via HF cache)

• Or run without transformers using Ollama (no Python ML deps):

  • Install and run Ollama; pull a vision model (e.g., ollama pull qwen2.5-vl)

  • Use backend ollama and set models in the config (e.g., caption_model: "qwen2.5-vl")

  • CLI example (triple, fully local):

    • python cli/image_metadata.py --image-url https://... --mode triple --caption-backend ollama --metadata-vision-backend ollama --config-path ./configs/triple_ollama_qwen.json

  • Configure host with --ollama-host http://localhost:11434 if not default

Per-call overrides (CLI)

• Metadata CLI now supports per-call backend overrides without editing global config:

  • --caption-backend local|openrouter|ollama (legacy: --ac-backend)

  • --metadata-vision-backend local|openrouter|ollama (legacy: --meta-vision-backend)

  • --local-vlm-id Qwen/Qwen2.5-VL-7B-Instruct (legacy: --local-model-id)

  • --ollama-host http://localhost:11434

Justfile tasks

• A Justfile provides quick test scenarios. Use URL-only inputs, e.g. just double_flash https://example.com/img.jpg.

• Scenarios included:

  • double_flash: Gemini 2.5 Flash for both steps

  • double_pro: Gemini 2.5 Pro for both steps

  • double_mixed_pro_text: Flash for vision alt+caption, Pro for text metadata (recommended mix for JSON reliability)

  • triple_flash / triple_pro: Flash/Pro for both vision steps

  • double_qwen_local <url> <qwen_id>: Local Qwen 2.5 VL for vision step, Pro for text metadata

  • triple_qwen_local <url> <qwen_id>: Fully local Qwen 2.5 VL for both vision steps

  • Convenience (no extra args):

    • double_qwen2b_local <url> / triple_qwen2b_local <url>

    • double_qwen7b_local <url> / triple_qwen7b_local <url>

Recommendation for mixed double

• Put Gemini 2.5 Pro on the text metadata step and Flash on the vision alt+caption step. The metadata step benefits from better structured-JSON compliance and reasoning, while Flash keeps latency/cost down for the vision caption.

• OpenRouter key requirements:

  • Double mode always requires OPENROUTER_API_KEY (text LLM for metadata).

  • Triple mode requires OPENROUTER_API_KEY unless both --ac-backend local and --meta-vision-backend local are set.

Examples

• MCP tool (local): {"backend": "local", "file_path": "./image.jpg"}

• CLI (local): python cli/caption_image.py --file-path ./image.jpg --backend local

Troubleshooting

• 401/403 from OpenRouter: ensure OPENROUTER_API_KEY is set and valid.

• Model selection: prefer cv_mcp.config.json at project root; or pass --config-path.

• Large images: remote images are downloaded and sent as base64; ensure the URL is accessible.

• Local backend: install optional deps pip install .[local] and ensure model is present/cached.

Changelog

• See docs/CHANGELOG.md for notable changes and release notes.