Computer Vision MCP Server

# cv-mcp Minimal MCP server focused on computer vision: image recognition and metadata generation via OpenRouter (Gemini 2.5 family). Goals - Keep it tiny and composable - Single tool: caption an image via URL or local file - No DB or app logic Structure - `src/cv_mcp/captioning/openrouter_client.py` – image analysis client - `src/cv_mcp/metadata/` – prompts, JSON schema, and pipeline runner - `src/cv_mcp/mcp_server.py` – MCP server exposing tools - `cli/caption_image.py` – optional CLI to test captioning locally Env vars - `OPENROUTER_API_KEY` Dotenv - Put `OPENROUTER_API_KEY` in a local `.env` file (see `.env.example`). - CLI scripts and the MCP server auto-load `.env` if present. Install - `pip install -e .` (or `pip install .`) ⚠️ **Development Note**: If you have the package installed via `pip install`, uninstall it before working with the local development version to avoid import conflicts. Use `pip uninstall cv-mcp` first, then run commands directly from the repo directory. Run MCP server (stdio) - Console script: `cv-mcp-server` (provides an MCP stdio server) - Configure your MCP client to launch `cv-mcp-server`. MCP integration (Claude Desktop) - Add to Claude Desktop config (see their docs for the config location): { "mcpServers": { "cv-mcp": { "command": "cv-mcp-server", "env": { "OPENROUTER_API_KEY": "sk-or-..." } } } } - After saving, restart Claude Desktop and enable the tool. Tools - `caption_image`: one-off caption (kept for compatibility) - `alt_text`: short alt text (<= 20 words) - `dense_caption`: detailed 2–6 sentence caption - `image_metadata`: structured JSON metadata with alt + caption. Params: - `mode`: `double` (default) uses 2 calls: vision (alt+caption) + text-only (metadata). `triple` uses vision for both steps. - `caption_override`: supply your own dense caption; skips the vision caption step. MCP tool reference - Server: `cv-mcp` (stdio) - `caption_image(image_url|file_path, prompt?, backend?, local_model_id?) -> string` - `alt_text(image_url|file_path, max_words?) -> string` - `dense_caption(image_url|file_path) -> string` - `image_metadata(image_url|file_path, caption_override?, config_path?) -> { alt_text, caption, metadata }` Examples - MCP call (OpenRouter): {"image_url": "https://example.com/image.jpg"} - MCP call (local): {"backend": "local", "file_path": "./image.jpg"} Quick test (CLI) - URL: `python cli/caption_image.py --image-url https://example.com/img.jpg` - File: `python cli/caption_image.py --file-path ./image.png` Metadata pipeline (CLI) - Double (default): - `python cli/image_metadata.py --image-url https://example.com/img.jpg --mode double` - Local alt+caption (still requires OpenRouter for metadata): - `python cli/image_metadata.py --image-url https://example.com/img.jpg --mode double --ac-backend local` - Triple (vision metadata): - `python cli/image_metadata.py --image-url https://example.com/img.jpg --mode triple` - Fully local (no OpenRouter required): - `python cli/image_metadata.py --image-url https://example.com/img.jpg --mode triple --ac-backend local --meta-vision-backend local` - With existing caption (skips the caption step): - `python cli/image_metadata.py --image-url https://example.com/img.jpg --caption-override "<dense caption>" --mode double` - Custom model config (JSON with `caption_model`, `metadata_text_model`, `metadata_vision_model`): - `python cli/image_metadata.py --image-url https://example.com/img.jpg --config-path ./my_models.json --mode double` Schema & vocab - JSON schema (lean): `src/cv_mcp/metadata/schema.json` - Controlled vocab (non-binding reference): `src/cv_mcp/metadata/vocab.json` Global config - Root file: `cv_mcp.config.json` (auto-detected from project root / CWD) - Env override: set `CV_MCP_CONFIG=/path/to/config.json` - Keys (renamed for clarity): - `caption_model`: vision model for alt+caption (OpenRouter) - `metadata_text_model`: text model for metadata (double mode) - `metadata_vision_model`: vision model for metadata (triple mode) - `caption_backend`: `openrouter` (default) or `local` for alt/dense/AC steps - `metadata_vision_backend`: `openrouter` (default) or `local` for triple mode - `local_vlm_id`: default local VLM (e.g. `Qwen/Qwen2.5-VL-7B-Instruct`) - Backwards-compat: legacy keys (`ac_model`, `meta_text_model`, `meta_vision_model`, `ac_backend`, `meta_vision_backend`, `local_model_id`) are still accepted. - Packaged defaults still live at `src/cv_mcp/metadata/config.json` and are used if no root config is found. - You can still provide a custom config file per-call via `--config-path` or the `config_path` tool param. Local backends (optional) - Install optional deps: `pip install .[local]` - Global default: set `"caption_backend": "local"` (and optionally `"metadata_vision_backend": "local"`) in `cv_mcp.config.json` - Use with MCP: pass `backend: "local"` in the tool params (overrides global) - Use with CLI: add `--backend local` and optionally `--local-model-id Qwen/Qwen2-VL-2B-Instruct` (overrides global) - Requires a locally available model (default: `Qwen/Qwen2-VL-2B-Instruct` via HF cache) - Or run without transformers using Ollama (no Python ML deps): - Install and run Ollama; pull a vision model (e.g., `ollama pull qwen2.5-vl`) - Use backend `ollama` and set models in the config (e.g., `caption_model: "qwen2.5-vl"`) - CLI example (triple, fully local): - `python cli/image_metadata.py --image-url https://... --mode triple --caption-backend ollama --metadata-vision-backend ollama --config-path ./configs/triple_ollama_qwen.json` - Configure host with `--ollama-host http://localhost:11434` if not default Per-call overrides (CLI) - Metadata CLI now supports per-call backend overrides without editing global config: - `--caption-backend local|openrouter|ollama` (legacy: `--ac-backend`) - `--metadata-vision-backend local|openrouter|ollama` (legacy: `--meta-vision-backend`) - `--local-vlm-id Qwen/Qwen2.5-VL-7B-Instruct` (legacy: `--local-model-id`) - `--ollama-host http://localhost:11434` Justfile tasks - A `Justfile` provides quick test scenarios. Use URL-only inputs, e.g. `just double_flash https://example.com/img.jpg`. - Scenarios included: - `double_flash`: Gemini 2.5 Flash for both steps - `double_pro`: Gemini 2.5 Pro for both steps - `double_mixed_pro_text`: Flash for vision alt+caption, Pro for text metadata (recommended mix for JSON reliability) - `triple_flash` / `triple_pro`: Flash/Pro for both vision steps - `double_qwen_local <url> <qwen_id>`: Local Qwen 2.5 VL for vision step, Pro for text metadata - `triple_qwen_local <url> <qwen_id>`: Fully local Qwen 2.5 VL for both vision steps - Convenience (no extra args): - `double_qwen2b_local <url>` / `triple_qwen2b_local <url>` - `double_qwen7b_local <url>` / `triple_qwen7b_local <url>` Recommendation for mixed double - Put Gemini 2.5 Pro on the text metadata step and Flash on the vision alt+caption step. The metadata step benefits from better structured-JSON compliance and reasoning, while Flash keeps latency/cost down for the vision caption. - OpenRouter key requirements: - Double mode always requires `OPENROUTER_API_KEY` (text LLM for metadata). - Triple mode requires `OPENROUTER_API_KEY` unless both `--ac-backend local` and `--meta-vision-backend local` are set. Examples - MCP tool (local): `{"backend": "local", "file_path": "./image.jpg"}` - CLI (local): `python cli/caption_image.py --file-path ./image.jpg --backend local` Troubleshooting - 401/403 from OpenRouter: ensure `OPENROUTER_API_KEY` is set and valid. - Model selection: prefer `cv_mcp.config.json` at project root; or pass `--config-path`. - Large images: remote images are downloaded and sent as base64; ensure the URL is accessible. - Local backend: install optional deps `pip install .[local]` and ensure model is present/cached. Changelog - See `docs/CHANGELOG.md` for notable changes and release notes.