# videre-mcp

[![CI](https://github.com/Veedubin/Videre-MCP/actions/workflows/ci.yml/badge.svg)](https://github.com/Veedubin/Videre-MCP/actions/workflows/ci.yml)
[![Python 3.11+](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Code style: ruff](https://img.shields.io/badge/code%20style-ruff-261230.svg)](https://docs.astral.sh/ruff/)

MCP server that bridges vision models to text-only coding models using Florence-2.

Non-vision LLMs can't see images — videre-mcp fixes that. It loads a Florence-2 vision model locally and exposes six MCP tools that convert images (including SVGs) and screenshots into structured text descriptions that any text-based model can consume.

```
Screenshot tool → videre-mcp (Florence-2) → Text description → Coding model
```

## Installation

```bash
pip install videre-mcp
```

Or with uv:

```bash
uv pip install videre-mcp
```

Requires Python 3.11+ and ~300MB disk space for the Florence-2-base model weights (downloaded automatically on first use).

## Usage

Add to your OpenCode configuration:

```json
{
  "mcpServers": {
    "videre-mcp": {
      "command": "videre-mcp"
    }
  }
}
```

Or run directly:

```bash
videre-mcp
# or
python -m videre_mcp
```

## Model Modes

All Florence-2 tools support a `model_mode` parameter to balance speed and quality:

- `"fast"` (default) — Uses **Florence-2-base**. Fast, lightweight, runs on CPU/GPU.
- `"deep"` — Uses **MiniCPM-V 4.6**. Significantly higher quality for complex visual reasoning.
  - **Requires:** `pip install videre-mcp[deep]`
  - **Hardware:** ~8GB VRAM recommended.

## Tools


### `describe_image`

Generate a natural language description of an image.

**Parameters:**
- `image_path` (str) — Path to the image file (supports PNG, JPEG, SVG)
- `detail_level` (str, optional) — `"normal"` (default) for brief caption, `"high"` for detailed description
- `model_mode` (str, optional) — `"fast"` (default) or `"deep"`


**Example:**

```python
result = describe_image("/path/to/photo.png", detail_level="high")
# Returns:
# {
#   "description": "A sunlit meadow with wildflowers in bloom...",
#   "model": "Florence-2-base",
#   "prompt_used": "<MORE_DETAILED_CAPTION>"
# }
```

### `ocr_image`

Extract text from an image using optical character recognition.

**Parameters:**
- `image_path` (str) — Path to the image file (supports PNG, JPEG, SVG)
- `detail_level` (str, optional) — `"normal"` (default) for plain text, `"high"` for text with bounding regions
- `model_mode` (str, optional) — `"fast"` (default) or `"deep"`


**Example:**

```python
result = ocr_image("/path/to/document.png", detail_level="high")
# Returns:
# {
#   "text": "Invoice Number 12345",
#   "regions": [
#     {"label": "Invoice Number 12345", "bbox": [10, 20, 30, 40, 50, 60, 70, 80]}
#   ]
# }
```

### `describe_screenshot`

Describe UI regions in a screenshot — designed for coding agents that need to understand screen layouts.

**Parameters:**
- `image_path` (str) — Path to the screenshot file (supports PNG, JPEG, SVG)
- `detail_level` (str, optional) — `"normal"` (default) for dense region captions, `"high"` for per-region descriptions
- `model_mode` (str, optional) — `"fast"` (default) or `"deep"`


**Example:**

```python
result = describe_screenshot("/path/to/screenshot.png")
# Returns:
# {
#   "regions": [
#     {"bbox": [10, 20, 30, 40], "label": "search bar"},
#     {"bbox": [100, 200, 300, 250], "label": "submit button"}
#   ],
#   "model": "Florence-2-base"
# }
```

### `take_screenshot`

Capture a screenshot and optionally describe it using Florence-2. Supports multi-monitor setups via the `monitor` parameter.

**Parameters:**
- `output_path` (str, optional) — Path to save the screenshot PNG. If `None`, saves to a temp file.
- `monitor` (int, optional) — Monitor index: `0` = all monitors combined, `1` = primary, etc. (default: `0`)
- `describe` (bool, optional) — If `True`, also run `describe_screenshot` on the captured image (default: `True`)
- `model_mode` (str, optional) — `"fast"` (default) or `"deep"`


**Example:**

```python
result = take_screenshot(monitor=1, describe=True)
# Returns:
# {
#   "path": "/tmp/tmpxxxxxx.png",
#   "width": 1920,
#   "height": 1080,
#   "monitor": 1,
#   "regions": [
#     {"label": "search bar", "bbox": [10, 20, 30, 40]},
#     ...
#   ]
# }
```

### `ocr_paddle`
Dedicated OCR using PaddleOCR (100+ languages, PP-OCRv6). Superior accuracy for multi-language documents.

**Parameters:**
- `image_path` (str) — Path to the image file
- `language` (str, optional) — Language code: `"en"`, `"ch"`, `"japan"`, `"korean"`, `"french"`, `"german"`, `"spanish"`, `"arabic"`, `"multilingual"`, etc. (default: `"en"`)
- `detail_level` (str, optional) — `"normal"` for plain text, `"high"` for text with bounding boxes and confidence scores
- `use_angle_cls` (bool, optional) — Use angle classification to correct rotated text (default: `True`)

**Requires:** `pip install videre-mcp[paddle]`

**Example:**
```python
result = ocr_paddle("/path/to/document.png", language="multilingual", detail_level="high")
# Returns:
# {
#   "text": "Invoice Number 12345\nDate: 2024-01-15",
#   "regions": [
#     {"text": "Invoice Number 12345", "bbox": [...], "confidence": 0.98}
#   ]
# }
```

### `parse_document`
Parse documents (PDF, DOCX, PPTX, HTML, MD) into structured output using IBM Docling. Extracts text, tables, charts, formulas, and code blocks.

**Parameters:**
- `file_path` (str) — Path to the document file
- `output_format` (str, optional) — `"markdown"` (default), `"json"`, `"text"`, or `"html"`
- `extract_tables` (bool, optional) — Extract and structure tables (default: `True`)
- `extract_images` (bool, optional) — Extract embedded images (default: `False`)

**Requires:** `pip install videre-mcp[docling]`

**Example:**
```python
result = parse_document("/path/to/report.pdf", output_format="markdown", extract_tables=True)
# Returns:
# {
#   "content": "# Report Title\n\n...",
#   "metadata": {"title": "...", "author": "...", "pages": 10},
#   "tables": [...]
# }
```

## Optional Dependencies

| Extra | Package | Enables |
|-------|---------|---------|
| `[deep]` | accelerate, bitsandbytes | MiniCPM-V 4.6 deep mode (~8GB VRAM) |
| `[docling]` | docling>=2.0.0 | Document parsing (PDF, DOCX, PPTX, HTML, MD) |
| `[paddle]` | paddleocr>=2.8.0 | PaddleOCR (100+ languages) |
| `[optimize]` | dspy-ai>=2.5.0 | DSPy prompt optimization CLI |

Install with: `pip install videre-mcp[deep,docling]`

## Requirements

- Python 3.11+
- ~300MB disk for model weights (auto-downloaded on first inference)
- Works on CPU; GPU (CUDA) is auto-detected and used if available

## Continuous Integration

The Florence-2 slow tests (real model load + inference) run on a nightly
schedule via GitHub Actions. See `.github/workflows/slow-tests.yml`.

## License

MIT — see [LICENSE](LICENSE).

### Third-party licenses

This package vendors a patched copy of Microsoft's Florence-2 processor
(`src/videre_mcp/_vendor/processing_florence2.py`) under Microsoft's MIT license.
See [src/videre_mcp/_vendor/LICENSE-Microsoft-Florence-2](src/videre_mcp/_vendor/LICENSE-Microsoft-Florence-2).