# Hybrid Vision MCP Server

A Model Context Protocol (MCP) server that exposes vision capabilities over HTTP, SSE, and Streamable HTTP transports. It bridges local vision engines—Tesseract.js (WASM OCR) and Sharp (image preprocessing)—with local Ollama vision models for image analysis, comparison, text localization, and browser screenshot analysis/annotation.

## Features

- **MCP Standard Compliance**: Exposes tools via `@modelcontextprotocol/sdk` v1.29.0 using Streamable HTTP and SSE transports.
- **Multi-Transport Support**: Works with `/mcp` (streamable HTTP), `/sse` (legacy SSE), and `/messages` endpoints.
- **Local-First Vision Stack**:
  - **OCR**: Fast CPU-based Tesseract.js WebAssembly engine (shipped with `eng.traineddata`).
  - **Preprocessing**: Sharp-powered crop, grayscale, and sharpen filters with boundary validation; returns full Base64 PNG via proper MCP `image` content blocks.
  - **AI Analysis**: Local Ollama vision models for description, comparison, element localization, rich browser screenshot analysis, visual diff, UI element detection, textual visual feedback generation, and semantic page extraction.
  - **Annotation Engine**: SVG-based overlay system for rendering labels, bounding boxes, arrows, and circles on images, returned as annotated PNGs.
  - **Repository Analysis**: Structural repo mapping via Graphviz DOT and JSON outputs for deep codebase understanding.
- **Accuracy Hardening**: Deterministic measurement (`measure_image`), schema-constrained and cross-validated analysis (`analyze_image_structured`), WCAG contrast maths, OCR cross-validation of model-read text, and full provenance on every response. See [`ACCURACY.md`](ACCURACY.md).
- **Flexible Image Input**: Accepts Base64 Data URIs, HTTP(S) URLs, `file://` URIs, local filesystem paths, `upload://` references, and direct file uploads via `/upload`.
- **Structured Responses**: Tools return JSON metadata in text blocks and full image data in `image` content blocks per MCP spec. Large output images are saved to disk and referenced by path to reduce payload bloat.
- **Hardened Upload Endpoint**: Binary image upload via `/upload` with MIME validation, magic-number verification, rate limiting, CORS support, and automatic cleanup of stale files.
- **Client Automation**: `upload-helper.js` automates the upload-then-reference flow for programmatic MCP clients.

## Prerequisites

- **Node.js** >= 18.x (ESM required)
- **Ollama** running locally (or reachable via `OLLAMA_HOST`)
- At least one vision model pulled in Ollama, e.g.:
  ```bash
  ollama pull llava:13b
  ollama pull qwen3-vl:30b
  ```

## Installation

```bash
git clone <repository-url>
cd hybrid-vision-mcp
npm install
```

## Environment Variables

The server loads configuration from environment variables. You can set them in three ways (listed in precedence order, highest first):

1. **Shell environment** — exported before `node index.js`
2. **`.env` file** — a file named `.env` in the project root is loaded automatically at startup (values only apply if not already set in the shell)
3. **Code defaults** — hardcoded fallbacks in `index.js`

Copy `.env.example` to `.env` and edit:

```bash
cp .env.example .env
```

| Variable | Default | Description |
|----------|---------|-------------|
| `PORT` | `11402` | Express listener port. |
| `OLLAMA_HOST` | `http://localhost:11434` | Base URL for the local Ollama inference service. |
| `OLLAMA_TIMEOUT_MS` | `180000` | Timeout for Ollama vision requests in milliseconds. |
| `MAX_PARALLEL_OLLAMA_REQUESTS` | `1` | Maximum number of Ollama vision requests allowed to run concurrently. Extra calls queue in-process rather than hitting Ollama simultaneously — vision models are GPU/unified-memory bound and concurrent inference has been observed to crash the host. |
| `MAX_OLLAMA_QUEUE_SIZE` | `20` | Maximum number of requests allowed to wait for a free Ollama slot before new ones are rejected outright. |
| `MCP_REQUEST_TIMEOUT_MS` | `300000` | Timeout for MCP client request/response in milliseconds. Heavy vision tools may exceed the default 60s SDK timeout; set this to match or exceed `OLLAMA_TIMEOUT_MS`. |
| `VISION_MODEL_FAST` | `llava:13b` | Default vision model for `analyze_image` and `detect_ui_elements`. |
| `VISION_MODEL_HEAVY` | `qwen3-vl:30b` | Default vision model for `find_text_element`, `compare_images`, `browser_screenshot_analysis`, `visual_diff`, and `detect_ui_elements`. |
| `UPLOAD_DIR` | `/tmp/hvm-uploads` | Directory for temporary binary image uploads. |
| `FEEDBACK_DIR` | `/tmp/hvm-feedback` | Directory for saved output images (annotations, diffs, overlays). |
| `DOWNLOAD_DIR` | `./tmp/hvm-downloads` | Directory for downloaded images from URLs (via `download_image` tool or transparent URL resolution). |
| `MAX_UPLOAD_SIZE_MB` | `20` | Maximum upload size in megabytes. |
| `MAX_DOWNLOAD_SIZE_MB` | `50` | Maximum download size in megabytes for URL-based image downloads. |
| `UPLOAD_RATE_LIMIT` | `10` | Maximum upload requests per minute per client IP. |
| `CORS_ORIGINS` | *(empty = allow all)* | Comma-separated list of allowed CORS origins for the `/upload` endpoint. |
| `ALLOWED_REPO_ROOTS` | *(server's own project directory)* | Comma-separated list of absolute directory paths that `generate_repo_graph` is permitted to walk. Prevents unauthenticated clients from enumerating the entire host filesystem. |

## Running the Server

```bash
# With .env file (recommended)
cp .env.example .env
# edit .env to suit your setup, then:
node index.js

# Or with inline environment variables:
PORT=11402 VISION_MODEL_FAST="llava:13b" VISION_MODEL_HEAVY="qwen3-vl:30b" node index.js
```

The server listens on `0.0.0.0` and exposes:

- **Health**: `GET http://localhost:<PORT>/health`
- **Streamable HTTP (MCP)**: `POST http://localhost:<PORT>/mcp`
- **Legacy SSE**: `GET http://localhost:<PORT>/sse` + `POST http://localhost:<PORT>/messages`
- **Image Upload**: `POST http://localhost:<PORT>/upload` (binary body, returns `upload://<filename>` reference)

If the default port `11402` is occupied by another service, set a custom port via the `PORT` environment variable:

```bash
PORT=3001 node index.js
```

Then point your MCP client configuration to the same port.

## Direct Image Upload Workflow

To avoid the ~33% size overhead and token cost of Base64 encoding, upload images directly to the `/upload` endpoint and pass the returned `upload://` reference to any tool.

### Step 1: Upload the image

```bash
curl -X POST http://localhost:11402/upload \
  --data-binary @screenshot.png \
  -H "Content-Type: image/png"
```

**Response**:
```json
{
  "uploadRef": "upload://<uuid>.png",
  "filename": "<uuid>.png",
  "mimeType": "image/png",
  "size": 5242880,
  "width": 1920,
  "height": 1080
}
```

### Step 2: Pass the reference to a tool

```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "analyze_image",
    "arguments": {
      "image_source": "upload://<uuid>.png",
      "prompt": "Describe this screenshot."
    }
  }
}
```

### Step 3: Retrieve output images from disk

Tools that produce large output images (`preprocess_and_crop`, `browser_screenshot_annotation`, `detect_ui_elements`, `visual_diff`) save results to `FEEDBACK_DIR` and include an `output_file_path` (or `diff_file_path`, `overlay_file_path`) in the JSON response. Clients can read the binary directly instead of parsing Base64 from the MCP response.

### Upload Rules

- **Max size**: 20 MB (configurable via `MAX_UPLOAD_SIZE_MB`).
- **Allowed types**: PNG, JPEG, WEBP, GIF, BMP.
- **Validation**: Magic-number verification ensures the file content matches the declared MIME type.
- **Rate limit**: 10 uploads per minute per IP (configurable via `UPLOAD_RATE_LIMIT`).
- **Cleanup**: Uploads are automatically deleted after 15 minutes. Feedback files follow the same TTL.

## MCP SDK Request Timeout

Heavy vision tools (`analyze_image`, `browser_screenshot_analysis`, `detect_ui_elements`, `find_text_element`, `compare_images`) rely on local Ollama vision models that may take longer than the MCP SDK's default 60-second request timeout to process large images.

**Symptom**: MCP error `-32001: Request timed out` when calling heavy vision tools.

**Solution**: Pass a `timeout` option to `client.callTool()` that matches or exceeds `OLLAMA_TIMEOUT_MS`:

```javascript
const result = await client.callTool(
  { name: "analyze_image", arguments: { image_source: url, prompt: "..." } },
  undefined,
  { timeout: 300000 }  // 5 minutes
);
```

The `upload-helper.js` library accepts an optional `requestTimeout` parameter in its constructor. When set, it automatically passes the timeout to every `callTool` and `callToolConnected` invocation:

```javascript
const helper = new MCPUploadHelper("http://localhost:11402", 300000);
```

The `MCP_REQUEST_TIMEOUT_MS` environment variable can be used to configure this value. See the [Environment Variables](#environment-variables) table above.

## Ollama Request Queuing

All tools that call Ollama (`analyze_image`, `find_text_element`, `compare_images`, `browser_screenshot_analysis`, `detect_ui_elements`, `visual_diff`) share a single in-process concurrency gate. Vision models are GPU/unified-memory bound, and running several inference calls in parallel has been observed to crash the host — so by default (`MAX_PARALLEL_OLLAMA_REQUESTS=1`) only one Ollama request runs at a time; additional calls wait in an in-process queue rather than being sent to Ollama concurrently.

- Increase `MAX_PARALLEL_OLLAMA_REQUESTS` only if the host has confirmed headroom to run more than one vision model inference simultaneously.
- `MAX_OLLAMA_QUEUE_SIZE` (default `20`) caps how many callers may wait for a slot; once exceeded, new requests fail fast with a clear error instead of piling up indefinitely.
- Waiting for a queue slot happens before the per-request Ollama timeout clock starts, so a busy queue doesn't erode a caller's `OLLAMA_TIMEOUT_MS` budget — but a client-side MCP request timeout (`MCP_REQUEST_TIMEOUT_MS` / `timeout` in `callTool()`) still applies across the whole wait-plus-inference duration.
- `check_vision_health` reports current queue depth (`Ollama Concurrency: X/Y active, Z queued`).

## File Input Schema Annotations

All image-accepting tools now declare their input fields with `x-mcp-file` annotations in the tool schema. This signals to MCP clients that the field accepts file uploads, enabling native file-picker UI where supported.

Example schema excerpt for `analyze_image`:

```json
{
  "name": "analyze_image",
  "inputSchema": {
    "type": "object",
    "properties": {
      "image_source": {
        "type": "string",
        "description": "Image to analyze. Accepts: Base64 Data URI, HTTP URL, local file path, or upload://<filename>.",
        "x-mcp-file": true,
        "x-mcp-file-accept": ["image/png", "image/jpeg", "image/webp", "image/gif"]
      }
    },
    "required": ["image_source"]
  }
}
```

Tools annotated with `x-mcp-file`:
- `fast_ocr_tesseract`
- `preprocess_and_crop`
- `analyze_image`
- `find_text_element`
- `compare_images`
- `browser_screenshot_analysis`
- `browser_screenshot_annotation`
- `detect_ui_elements`
- `visual_diff`
- `textual_visual_feedback`

> **Note**: `x-mcp-file` is a non-standard extension aligned with the MCP File Uploads Working Group's SEP-2356 direction. Existing clients ignore unknown schema properties, so this is fully backward-compatible.
## Client Helper Library

`upload-helper.js` automates the upload-then-reference flow for Node.js MCP clients. It detects `File`, `Blob`, `ArrayBuffer`, `Buffer`, or string inputs; uploads binary data to `/upload`; and replaces the argument with an `upload://` reference before calling the tool.

### Usage

```javascript
import { MCPUploadHelper } from "./upload-helper.js";

const helper = new MCPUploadHelper("http://localhost:11402");

// One-shot tool call with automatic upload
const result = await helper.callTool("analyze_image", {
  image_source: "/path/to/local/screenshot.png",
  prompt: "Describe this screenshot.",
});
console.log(result.content[0].text);

// Batch upload for multi-image tools
const comparison = await helper.callTool("compare_images", {
  image_sources: ["/tmp/before.png", "/tmp/after.png"],
  prompt: "List all visual differences.",
});

// Reusable connected session
const client = await helper.connect();
await helper.callToolConnected("fast_ocr_tesseract", {
  image_source: "data:image/png;base64,...",
  language: "eng",
});
await helper.close();
```

See `client-example.js` for runnable demonstrations.

## Client Configuration

For MCP clients that use a `config.json` format (e.g., VS Code or Kilo MCP extensions), add:

```json
{
  "mcpServers": {
    "hybrid-vision": {
      "url": "http://localhost:11402/sse"
    }
  }
}
```

## Available Tools

### 1. `check_vision_health`

Check connectivity to the local Ollama service and verify installed vision engines.

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| *(none)* | | | No parameters required. |

**Returns**: Text report with Ollama connection status, installed models, and engine readiness.

---

### 2. `fast_ocr_tesseract`

Fast CPU-based WebAssembly OCR extraction for text in images.

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `image_source` | string | **Yes** | — | Base64 Data URI, HTTP URL, local path, or `upload://<filename>`. |
| `language` | string | No | `"eng"` | ISO 639-3 language code (e.g. `"spa"`, `"fra"`). |

**Returns**: Extracted text with confidence score.

---

### 3. `preprocess_and_crop`

Computer vision image preprocessing using Sharp.

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `image_source` | string | **Yes** | — | Base64 Data URI, HTTP URL, local path, or `upload://<filename>`. |
| `crop` | object | No | — | Region to extract: `{ left, top, width, height }`. |
| `grayscale` | boolean | No | — | Apply grayscale filter. |
| `sharpen` | boolean | No | — | Apply sharpen filter. |

**Returns**: JSON metadata with `output_file_path` pointing to the saved PNG, plus an `image` content block with Base64 data.

**Crop Validation**:
- `left` and `top` must be `>= 0`.
- `width` and `height` must be `> 0`.
- Crop region must be fully contained within the image boundaries.

---

### 4. `analyze_image`

Analyze image contents or extract context using local Ollama vision models.

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `image_source` | string | **Yes** | — | Base64 Data URI, HTTP URL, local path, or `upload://<filename>`. |
| `prompt` | string | No | `"Describe this image in detail."` | Question or instruction for analyzing the image. |
| `model` | string | No | `VISION_MODEL_FAST` | Optional Ollama vision model override. |

**Returns**: Raw text response from the Ollama vision model.

---

### 5. `find_text_element`

Locate specific text, UI elements, or objects visually within an image.

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `image_source` | string | **Yes** | — | Base64 Data URI, HTTP URL, local path, or `upload://<filename>`. |
| `query` | string | **Yes** | — | Target text or element to locate. |
| `model` | string | No | `VISION_MODEL_HEAVY` | Optional Ollama vision model override. |

**Returns**: Text description of the bounding-box coordinates or visual position of the target element.

---

### 6. `compare_images`

Compare two or more images side-by-side using local Ollama vision models.

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `image_sources` | array[string] | **Yes** | — | Array of image inputs. Each accepts Base64 Data URI, HTTP URL, local path, or `upload://<filename>`. |
| `prompt` | string | No | `"Compare these images in detail and highlight any differences or similarities."` | Comparison instructions. |
| `model` | string | No | `VISION_MODEL_HEAVY` | Optional Ollama vision model override. |

> **CRITICAL**: `image_sources` MUST be a raw JSON array of strings. Do NOT pass a single stringified array.

**Returns**: Raw text response from the Ollama vision model comparing all provided images.

---

### 7. `browser_screenshot_analysis`

Perform high-level visual and semantic analysis of a browser screenshot or UI image. Generates a rich description of layout, components, visual hierarchy, colors, typography, spacing, and overall design "vibe" to help agents understand the current UI state without manual inspection.

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `image_source` | string | **Yes** | — | Base64 Data URI, HTTP URL, local path, or `upload://<filename>`. |
| `focus` | string | No | `"all"` | Analysis focus: `all`, `layout`, `components`, `accessibility`, `design`, or `content`. |
| `detail_level` | string | No | `"standard"` | Level of detail: `brief`, `standard`, or `detailed`. |
| `model` | string | No | `VISION_MODEL_HEAVY` | Optional Ollama vision model override. |

**Returns**: Structured text summary prefixed with analysis metadata.

---

### 8. `browser_screenshot_annotation`

Annotate a screenshot or image with text labels, bounding boxes, arrows, or circles to highlight specific UI components, regions of interest, or action targets.

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `image_source` | string | **Yes** | — | Base64 Data URI, HTTP URL, local path, or `upload://<filename>`. |
| `annotations` | array[object] | **Yes** | — | Array of annotation objects (see schema below). |
| `return_base64` | boolean | No | `true` | If true, appends an `image` content block with the full annotated Base64 PNG. If false, returns only a JSON metadata text block. |

**Annotation Object Schema**:

| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `type` | string | **Yes** | — | `label`, `box`, `arrow`, or `circle`. |
| `text` | string | Conditional | — | Text content for `label` annotations. |
| `x` | number | Conditional | — | X coordinate (pixels). |
| `y` | number | Conditional | — | Y coordinate (pixels). |
| `width` | number | Conditional | — | Width (box) or radius (circle) in pixels. |
| `height` | number | Conditional | — | Height (box) in pixels. |
| `target_x` | number | Conditional | — | Target X for arrow endpoint. |
| `target_y` | number | Conditional | — | Target Y for arrow endpoint. |
| `color` | string | No | `#FF0000` | Hex color code. |
| `font_size` | number | No | `16` | Font size in pixels for labels. |

**Returns** (default `return_base64=true`):
```json
{
  "content": [
    {
      "type": "text",
      "text": "{ \"success\": true, \"width\": 800, \"height\": 600, \"format\": \"image/png\", \"output_file_path\": \"/tmp/hvm-feedback/...\", \"data_uri_length\": 123456, \"data_uri\": \"data:image/png;base64,...\" }"
    },
    { "type": "image", "data": "<base64_png>", "mimeType": "image/png" }
  ]
}
```

---

### 9. `visual_diff`

Compare two screenshots and highlight visual changes between them. Generates a pixel-level diff image that colors changed regions, and optionally an AI-generated description of what differs between the `before` and `after` states.

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `image_sources` | array[2 strings] | **Yes** | — | Array of exactly 2 images: [before, after]. |
| `threshold` | number | No | `15` | Minimum combined RGB delta to mark a pixel as changed (0-255). Lower values catch subtle changes. |
| `highlight_color` | string | No | `#FF00FF` | Hex color for changed pixels in the diff image. |
| `analyze` | boolean | No | `true` | If true, includes an Ollama Vision description of the differences. |

**Returns**: JSON metadata with `diff_file_path` pointing to the saved PNG, plus an `image` content block with Base64 data.

---

### 10. `detect_ui_elements`

Detect UI components and interactive elements in a screenshot using a vision model. Returns structured element descriptions with approximate bounding boxes and an optional overlay visualization.

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `image_source` | string | **Yes** | — | Base64 Data URI, HTTP URL, local path, or `upload://<filename>`. |
| `element_types` | array[string] | No | all common UI elements | Filter to specific types: `button`, `input`, `link`, `card`, `navigation`, `modal`, `dropdown`, `checkbox`, `radio`, `table`, `list`, `icon`, `heading`. |
| `return_overlay` | boolean | No | `false` | If true, attempts to overlay detected elements as green bounding boxes on the image. |
| `model` | string | No | `VISION_MODEL_HEAVY` | Optional Ollama vision model override. |

**Returns** (default `return_overlay=false`):
```json
{
  "content": [
    {
      "type": "text",
      "text": "{ \"success\": true, \"width\": 800, \"height\": 600, \"detection\": \"Buttons: [x=...], Inputs: [x=...]\" }"
    }
  ]
}
```

**Returns** (when `return_overlay=true`):
```json
{
  "content": [
    {
      "type": "text",
      "text": "{ \"success\": true, \"width\": 800, \"height\": 600, \"detection\": \"...\", \"overlay_data_uri\": \"data:image/png;base64,...\", \"overlay_file_path\": \"/tmp/hvm-feedback/...\" }"
    },
    { "type": "image", "data": "<base64_png>", "mimeType": "image/png" }
  ]
}
```

---

### 11. `textual_visual_feedback`

Generate a concise feedback object in JSON format integrating a screenshot, DOM tree, CSS styles, and OCR-derived text data. The screenshot is saved to a local file to avoid context window overflow, and only metadata plus the file path are returned.

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `image_source` | string | **Yes** | — | Base64 Data URI, HTTP URL, local path, or `upload://<filename>`. |
| `dom_fragment` | string | No | `""` | Optional HTML DOM fragment as string to include in feedback. |
| `css_snapshot` | string | No | `""` | Optional CSS styles as string to include in feedback. |
| `include_ocr` | boolean | No | `true` | If true, run OCR on the screenshot to extract text. |
| `ocr_language` | string | No | `eng` | OCR language code (e.g. `eng`, `spa`). |

**Returns**:
```json
{
  "content": [
    {
      "type": "text",
      "text": "{ \"success\": true, \"timestamp\": \"2024-...\", \"screenshot\": { \"mime_type\": \"image/png\", \"width\": 800, \"height\": 600, \"data_uri_length\": 954504, \"file_path\": \"/tmp/hvm-feedback/1700000000000-abc.png\" }, \"ocr\": { \"enabled\": true, \"confidence\": 85, \"text\": \"...\" }, \"dom\": { \"provided\": true, \"fragment_length\": 42 }, \"css\": { \"provided\": false } }"
    }
  ]
}
```

---

### 12. `extract_semantic_page`

Extract structured semantic layout from HTML using DomDistiller-inspired algorithms. Produces a structured Control Map with headings, navigation, content blocks, forms, tables, and other semantic elements.

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `html_content` | string | **Yes** | — | HTML content as string to analyze. |
| `min_text_length` | number | No | `10` | Minimum character length for text blocks to include. |
| `include_raw` | boolean | No | `false` | If true, include raw text stats in output. |

**Returns**:
```json
{
  "content": [
    {
      "type": "text",
      "text": "{ \"success\": true, \"control_map\": { \"title\": \"...\", \"headings\": [...], \"navigation\": {...}, \"main_content\": [...], \"lists\": [...], \"forms\": [...], \"tables\": [...], \"media\": [...], \"footer\": {...} } }"
    }
  ]
}
```

---

### 13. `generate_repo_graph`

Generate repository structural map using Graphviz DOT and JSON formats. Analyzes file tree, classifies files by extension, builds dependency-like parent-child graph for deep codebase understanding.

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `repo_path` | string | **Yes** | — | Absolute path to repository root directory. Must be within `ALLOWED_REPO_ROOTS` (see [Environment Variables](#environment-variables)) or the call is rejected. |
| `max_depth` | number | No | `5` | Maximum directory depth to traverse. |
| `include_node_modules` | boolean | No | `false` | If true, include `node_modules` in traversal. |

**Returns**:
```json
{
  "content": [
    {
      "type": "text",
      "text": "{ \"success\": true, \"graph\": { \"node_count\": 42, \"edge_count\": 38, \"nodes\": [...], \"edges\": [...] } }"
    },
    {
      "type": "text",
      "text": "--- Graphviz DOT ---\ndigraph repo {\n  rankdir=TB;\n  ...\n}"
    }
  ]
}
```

---

### 14. `download_image`

Download an image from a URL into the server's `DOWNLOAD_DIR` and return a `download://` reference that can be passed to any other tool. Validates the URL, checks content-type, verifies image magic bytes, and enforces size limits.

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `url` | string | **Yes** | — | HTTP or HTTPS URL of the image to download. Hostname must resolve to a public IP address — loopback, RFC1918/link-local/CGNAT ranges (including cloud metadata endpoints) are rejected as an SSRF guard. This same check applies transparently whenever any tool's `image_source` is an HTTP(S) URL. |
| `filename` | string | No | auto-generated | Optional custom filename (without path). |

**Returns**:
```json
{
  "content": [
    {
      "type": "text",
      "text": "{ \"success\": true, \"downloadRef\": \"download://<uuid>.png\", \"file_path\": \"/abs/path/to/tmp/hvm-downloads/<uuid>.png\", \"original_url\": \"https://...\", \"mime_type\": \"image/png\", \"size\": 123456, \"width\": 800, \"height\": 600 }"
    }
  ]
}
```

**Error scenarios**:
- Invalid URL format (not http/https)
- DNS/network failure
- HTTP error (4xx/5xx)
- Wrong content-type (e.g., text/html)
- Payload does not contain valid image magic bytes
- Empty response
- File exceeds `MAX_DOWNLOAD_SIZE_MB`
```

---

### 15. `measure_image`

Answer **measurable** questions with code instead of a language model. WCAG
contrast, dominant/background colours, box counts, and region content are
computed from pixels using Sharp, so results are reproducible and available
even when Ollama is unavailable or contended. **No vision model is consulted.**

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `image_source` | string | **Yes** | — | Base64 Data URI, HTTP URL, local path, or `upload://<filename>`. |
| `mode` | string | No | `"all"` | `contrast`, `colors`, `boxes`, `layout`, or `all`. |
| `region` | object | No | — | Region of interest `{ left, top, width, height }`. **Required for meaningful contrast.** |
| `box_color` | string | No | — | Border colour (hex) of the boxes to count. Required for `boxes`. |
| `background_color` | string | No | — | Explicit background colour. If omitted, the region's modal colour is used. |
| `tolerance` | number | No | `24` | RGB distance tolerance for colour matching (0-255). |
| `large_text` | boolean | No | `false` | Assess against WCAG large-text thresholds (3:1). |
| `ink_threshold` | number | No | `4` | RGB distance from the background above which a pixel counts as ink. Lower it to catch text closer to the background. |
| `background_mode` | string | No | `"global"` | `"global"` = one modal background colour (correct for flat UI screenshots). `"local"` = per-tile robust background with a noise-scaled threshold, for photographic/gradient/multi-tone images. |
| `tile_size` | number | No | `48` | Tile edge in pixels for `background_mode: "local"`. |
| `noise_factor` | number | No | `4` | `k` in `threshold = max(ink_threshold, k * local_noise_sigma)` for local mode. |
| `include_decorative` | boolean | No | `false` | Include decorative (thin hollow rectangle) colours in the summary instead of reporting them separately. |
| `decorative_colors` | array[string] | No | — | Hex colours to treat as decorative explicitly. |

**Contrast enumerates every text colour and reports the worst case.** The old
behaviour selected only the most legible colour, which meant an image containing
large text at 2.14:1 and 1.04:1 could be summarised as `wcag_aa: true` — reading
as "no contrast problems". Now:
```json
"contrast": {
  "worst":  { "foreground": "#1e1c18", "contrast_ratio": 1.04, "wcag_aa": false },
  "best":   { "foreground": "#e8dfd0", "contrast_ratio": 13.42, "wcag_aa": true },
  "failing_count": 2,
  "passing_count": 3,
  "evaluated_count": 5,
  "all_meet_aa": false,
  "wcag_aa": false,
  "contrast_ratio": 1.04,
  "colours": [
    { "foreground": "#1e1c18", "pixel_count": 2241, "contrast_ratio": 1.04, "wcag_aa": false, "meets_aa": false },
    { "foreground": "#484f58", "pixel_count": 2792, "contrast_ratio": 2.14, "wcag_aa": false, "meets_aa": false },
    { "foreground": "#a09588", "pixel_count": 639,  "contrast_ratio": 6.03, "wcag_aa": true },
    { "foreground": "#7daa7a", "pixel_count": 1052, "contrast_ratio": 6.67, "wcag_aa": true },
    { "foreground": "#e8dfd0", "pixel_count": 4539, "contrast_ratio": 13.42, "wcag_aa": true }
  ],
  "excluded": [ { "foreground": "#4a433c", "contrast_ratio": 1.82, "reason": "decorative: thin hollow rectangle geometry (border/rule), not text" } ],
  "notes": ["...", "2 of 5 evaluated text colour(s) fail WCAG AA: ..."]
}
```

> **Semantics change:** `wcag_aa` at the contrast level now aliases the **worst**
> case (`all_meet_aa`) — `true` only when every evaluated text colour passes.
> `best.contrast_ratio` preserves the previous number, and `foreground` /
> `contrast_ratio` now refer to the worst case. See `ACCURACY.md` §5b.

**Nothing is silently dropped.** Excluded (decorative) and skipped (sub-threshold)
colours are listed with their ratio and pixel count and described in `notes`, and
they are excluded from `all_meet_aa`. If no text colour is assessable the result
is `measurable: false` with an `abstained[]` entry rather than a verdict. In
`background_mode: "local"`, clusters judged to be residual noise rather than text
appear under `suspected_noise` (also excluded from the verdict, also disclosed).

**Photographic and gradient images.** A single global background colour only
models flat UI screenshots well. When it explains less than half the region, the
result carries a `background_fit` block and a warning in `notes`:
```json
"background_fit": {
  "background": "#90877d",
  "explained_fraction": 0.007,
  "tolerance": 8,
  "adequate": false,
  "applicable": true,
  "warnings": ["A single global background colour accounts for only 0.7% ..."]
}
```

A fit between 0.5 and 0.8 is **marginal**: it clears the adequacy floor but is
still a weak model, so it is disclosed too, and a verdict is never silent about it.

The `adequate` flag is only meaningful when a single background applies. In a
multi-plateau region it is `null` (with `applicable: false` and a reason), so it
cannot read as "fine" for a value that does not apply there.

For such images pass `background_mode: "local"`, which uses a per-tile median
background with a threshold that scales with each tile's measured noise. On a
text-free photo that yields **zero** reported colours (rather than the whole
gradient counting as one), and it recovers text tones a global background misses.
The mode is opt-in because a tile straddling two flat panels can
over-threshold — measured, not assumed. See `ACCURACY.md` §5c.

**Multi-panel UIs (sidebar + content).** A single background colour cannot model
a UI with two or more large flat panels: the non-modal panel used to become one
enormous "ink" blob whose colour was the panel *fill*, and text sitting on that
panel disappeared entirely — so a 1.64:1 sidebar string could be reported as
`all_meet_aa: true`. When two or more large flat plateaus are detected, every
plateau is treated as background and **each text colour is measured against the
fill it actually sits on**:

```json
"background_model": "multi-plateau",
"plateaus": [ { "hex": "#d2d2d2", "share": 0.6928 }, { "hex": "#161616", "share": 0.2892 } ],
"colours": [
  { "foreground": "#3c3c3c", "contrast_ratio": 1.64, "wcag_aa": false,
    "measured_against": "#161616", "measured_against_plateau": "#161616", "local_background": "#161616" },
  { "foreground": "#282828", "contrast_ratio": 9.75, "wcag_aa": true,
    "measured_against": "#d2d2d2", "measured_against_plateau": "#d2d2d2" }
],
"panel_fills": [ { "foreground": "#d2d2d2", "reason": "panel fill: a large near-solid region ..." } ],
"notes": [ "Multi-plateau region: 2 large flat colour plates were detected ..." ]
```

This is structural, so it fires even when the modal colour explains most of the
area (a 30/70 layout scores `explained_fraction: 0.693`, which a single-colour
adequacy test would call fine). A plateau must pass four measured tests —
minimum region size, component dominance (a panel is ONE blob; large text is
many), flatness (a panel is one exact colour; a gradient band is not) and
connectivity. With fewer than two plateaus the result is unchanged. See
`ACCURACY.md` §5e.

**Tiled layouts are panels too.** A dashboard repeats its cards, so a card fill is
"many small identical blobs" — the same signature as a glyph run. A second,
structural plateau path recognises a repeated fill by **solidity and size per blob**
rather than blob count (>= 2 blobs, each >= 0.4% of the region, fill >= 0.85,
similar sizes). Without it the page background around the cards became one huge
component and was reported as failing text on an image where every colour passes.
`plateaus[].detection` says which path matched (`"dominant-blob"` or `"tiled"`).

```json
"plateaus": [
  { "hex": "#1a1814", "share": 0.3768, "detection": "dominant-blob" },
  { "hex": "#2d2822", "share": 0.6021, "detection": "tiled", "solid_component_count": 12 }
],
"background_model": "multi-plateau",
"colours": [ { "foreground": "#e8dfd0", "contrast_ratio": 11.05, "wcag_aa": true } ]
```

**A background region is never a failing text colour.** A near-background cluster
whose blobs are individually large is a region separated by panels, not a text run,
so it is listed under `background_regions[]` and excluded from the verdict (with a
note). Real text averages <= 0.10% of the region per blob against 37.7% for a page
background, so this cannot drop a text run. See `ACCURACY.md` §5g.

**A text colour is never a plateau.** A plateau blob must be either the outermost
colour (the background, with panels cut out of it, so a low fill is expected) or an
inset blob that is not a glyph RING. A ring is recognised by one large enclosed
aperture; a content-dense panel (a chart card, a table, a thumbnail grid) is
perforated by many small holes and is still a panel. Two huge identical glyphs can
no longer be mistaken for a background — which had made failing text disappear.
See `ACCURACY.md` §5h-§5i.

**Mask reconciliation.** When a plateau mask removed a colour, the region is
re-enumerated with no mask. Any failing colour the masked result dropped is
disclosed under `mask_reconciliation` and in `notes`, with the raw evidence:

```json
"mask_reconciliation": {
  "unmasked_failing_colours": [ {
    "foreground": "#464646", "contrast_ratio": 1.88,
    "component_count": 164, "mean_component_area": 270,
    "detected_plateau": true, "plateau_share": 0.0489, "largest_component_share": 0.0174
  } ],
  "note": "masking a background region also removed colour(s) that an un-masked pass measures as failing. These are usually backgrounds or decoration, but they could be text. The tool does NOT classify which: judge from the reported evidence ..."
}
```

**The gate is UNCONDITIONAL.** Every failing colour the mask removed is disclosed —
there is no size threshold. The tool does **not** classify whether a flagged colour is
decoration or text; it reports raw evidence so you can judge:

- `detected_plateau` — whether the colour was itself read as a background plateau
  (evidence, **not** a classifier: real text is often read as a plateau too);
- `plateau_share` — the plateau's **coverage** of the region. Fragmentation-invariant,
  but it orders **large from small, not decoration from text**: a dense glyph run can
  cover MORE than a bar chart (measured 0.1561 text vs 0.0986 bars);
- `component_count`, `mean_component_area`, `largest_component_share`, `pixel_count`.

It remains disclosure, not refusal, so it cannot turn a correct verdict into a wrong
one.

> **History (three attempts to add a size gate, all reverted).** A gate looks like
> precision but keeps hiding real text:
> - **Tenth audit (F12):** `mean >= 0.4%` compared a mean to the mask's *single-blob*
>   floor, hiding a 2%-blob colour with many small companions.
> - **Eleventh audit (F13):** a union added `largestShare >= 2%` (true by construction
>   for the mask's path-A) but kept the mean clause.
> - **Twelfth audit (F14):** the mean clause was shown to be **anti-correlated with
>   the evidence** — adding six small body lines to a 300px failing heading took the
>   colour from disclosed (mean 1.78%) to silent (mean 0.03%) while the largest blob
>   was unchanged. **More failing text made the warning disappear.** The gate is
>   therefore removed, not re-tuned. See `ACCURACY.md` §5n.
>
> **Consequence:** `mask_reconciliation` now fires on decorative dashboards (the
> ninth-audit F11 case), because the un-masked pass re-reads a tiled bar colour as
> failing text. That is the deliberate trade — the disclosure is advisory and names
> its own evidence, whereas the F14 omission was silent. If you want to ignore
> decoration, filter on `detected_plateau` / `plateau_share` rather than expecting it
> to be suppressed. Be aware (fourteenth audit F16) that **neither field classifies**
> decoration vs text — see `ACCURACY.md` §5p.
>
> **No classification is provided (thirteenth audit F15).** An earlier version also
> emitted `shape: "panel-shaped" | "text-sized"`. It was **removed**: its threshold was
> `mean = total / N` again, so it flipped with fragmentation and inverted on a real
> pair (9 decorative bars and a 164-glyph failing run were *both* `text-sized`). The
> replacement the audit proposed — the largest connected component — was measured and
> **does not separate** them either (0.0185 vs 0.0174). A bar chart and a glyph run are
> the same kind of object, so no scalar is offered in its place. See `ACCURACY.md` §5o.

> **Documentation-guard scope (F16–F26).** This repository also carries a **lint** that
> scans the docs for prose asserting that an exposed field classifies decoration vs text.
> The lint **fails the build** when it sees a claim; what we no longer rely on is its
> **completeness**. Its measured recall on the fixture set is **18/19** (a regression score
> over the known escape cases, **not** an estimate over unseen prose; P4 is the known miss),
> printed by the test and enforced so it cannot silently drop. The **part we trust is the
> positive assertion**: the docs must **contain** the explicit disclaimer (they do). The
> acceptance test that matters is the **invariance pair** (a claim must get the same verdict
> with or without each waiver mechanism). See `ACCURACY.md` §5r–§5v.
>
> **The guard is FROZEN (twenty-first audit).** Following the round-19 recommendation, no
> further effort goes into extending this lint; later rounds spend their substance on the
> **measurement engine** instead. See `ACCURACY.md` §5w for that round's two engine defects
> (a text-free gradient reported as failing text; the darker of two colours in outlined text
> silently absorbed), §5x for the follow-up (F29): the outlined-text pixel floor was
> applied **per component**, so small glyphs lost their failing fill as it split across
> components — it is now applied to the colour's **total** across the scan; §5y for the
> next round (F30/F31): the second-ink path had its **own** 28×-larger floor, which hid
> failing ink and admitted accumulated card/text anti-aliasing fringes — it now uses the
> primary path's reporting floor and gates; §5z (F32): a further divergence on the
> **colour-proximity** axis let a real fill that lies on the reference→stroke line be
> discarded as a fringe — the AA test now rejects a blend only when it is **small**; §5aa (F34):
> that size qualifier was applied **per component**, so a real on-line fill split across glyphs
> was discarded — it is now applied to the colour's **on-line total** in the deferred pass. In
> the second-ink path, **no gate may use a per-component quantity without stating why the total
> is wrong**. §5ab (F35): the same per-piece shape appeared in the **primary** path's
> large-background-region gate (a split decoration was reported as failing text); the gate now
> requires the **total** *and* **solidity**, keeping the original mean test for the textured-page
> backstop. A standing **count-invariance** test (fixed total, varying piece count) now covers
> this class; its **coverage is measured** (§5ac) — it catches F32/F34/F35, and the cases it does
> *not* cover are stated rather than assumed.

---

**Returns** (excerpt):
```json
{
  "success": true,
  "mode": "contrast",
  "dimensions": { "width": 1200, "height": 760, "format": "png" },
  "measurements": {
    "contrast": {
      "background": { "hex": "#1a1814", "rgb": { "r": 26, "g": 24, "b": 20 } },
      "foreground": { "hex": "#484f58", "rgb": { "r": 72, "g": 79, "b": 88 } },
      "contrast_ratio": 2.14,
      "wcag_aa": false,
      "wcag_aaa": false,
      "required_aa": 4.5,
      "measurable": true,
      "method": "pixel-histogram/exact-colour-mode+max-luminance-delta"
    }
  },
  "abstained": [],
  "provenance": { "model": null, "image": { "sent_dimensions": { "width": 1200, "height": 760 }, "downscaled": false } },
  "disclaimer": "Deterministic measurement. Reproducible without a vision model..."
}
```

**Honest abstention**: `boxes` mode without `box_color` abstains (counting without
knowing the border colour would be a guess). `contrast` mode with no `region` does
**not** abstain — it measures the whole frame and says so in `notes`, exactly as
`mode: "all"` does, so the same question always gets the same answer regardless of
`mode`. Pass a `region` when the question is about specific text.

**Noise is not called out as text.** A weak cluster that is not shaped like text
(fewer than 3 components or a sub-100px mean component area) is returned under
`suspected_noise` and named as suspected noise in `notes`, rather than being
asserted as a failing text colour. It is excluded from
`worst`/`failing_count`/`all_meet_aa` but never hidden.

**Local mode has no single background.** Under `background_mode: "local"`,
`background` is `null` (there is no single background by construction, and the
global modal colour can coincide with a text tone).

**Every colour says what it was measured against.** Each `colours[]` entry carries
`measured_against` (the fill the text sits on) and `local_background`, so a caller
never has to infer which background produced a ratio.

**Panel fills are not text colours.** In a multi-plateau region the large
near-solid panel colours are returned under `panel_fills[]` and excluded from the
text enumeration, so a panel's own fill is never reported as a "failing colour".

**Straight strokes are chrome, not glyphs.** A component that is long and thin in
one axis (≥ 60px long, ≤ 4px across) — a rule, divider or card border — is treated
as decorative, so a bordered card grid is not reported as failing text.

**A clean verdict is never silent about a weak premise.** When the global model
reports no failing colour, the region is re-measured with the per-tile (local)
model. If local finds a failing tone the global model missed — the signature of a
background that varies across the region — the result carries a
`model_disagreement` block and a note asking you to re-measure with a `region`:

```json
"model_disagreement": {
  "global_all_meet_aa": true,
  "local_all_meet_aa": false,
  "local_failing_count": 1,
  "local_failing_colours": [ { "foreground": "#8c8c8c", "contrast_ratio": 3.81, "measured_against": "#3f3f3f" } ],
  "note": "the two background models disagree on this region ... Treat the clean verdict as UNVERIFIED ... Note that local mode is not generally more accurate: it over-reports on dense flat panels, which is why it is not the default."
}
```

This is a detector, not an endorsement: on dense flat panels local *over-reports*,
so the note explicitly does not claim local is better.

---

### 16. `analyze_image_structured`

Analyze an image and return **structured, evidence-backed** output instead of
free prose. Every statement is a `claims[]` entry carrying its own bounding box
and confidence; transcribed text is cross-validated against Tesseract OCR and
flagged as `unverified` when it has no OCR support and no plausible box (the
fabrication signature). Abstention is first-class. Quantitative questions are
answered by deterministic measurement in code, never by the model.

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `image_source` | string | **Yes** | — | Base64 Data URI, HTTP URL, local path, or `upload://<filename>`. |
| `prompt` | string | No | `""` | What to find out. A quantitative prompt is routed to measurement. |
| `model` | string | No | `VISION_MODEL_FAST` | Optional Ollama vision model override. |
| `crop` | object | No | — | `{ left, top, width, height }`. Physically applied before analysis. |
| `cross_validate` | boolean | No | `true` | Cross-validate transcribed text against Tesseract OCR. |
| `ocr_language` | string | No | `"eng"` | OCR language for cross-validation. |
| `timeout_ms` | number | No | `OLLAMA_TIMEOUT_MS` | Per-request Ollama timeout. |
| `keep_alive` | string | No | `OLLAMA_KEEP_ALIVE` | e.g. `"5m"`, or `"0"` to unload after the call. |
| `seed` | number | No | `42` | Sampling seed for reproducibility. |
| `temperature` | number | No | `0` | Sampling temperature. |

**Returns** (excerpt):
```json
{
  "success": true,
  "parsed": true,
  "summary": "...",
  "claims": [
    { "claim": "...", "kind": "text", "box": { "left": 40, "top": 40, "width": 200, "height": 40 },
      "confidence": 0.8, "source": "vl", "verified": false, "basis": "vision-model-observation" }
  ],
  "unsupported_claims": [],
  "text_items": [
    { "text": "ALPHA-ONE", "source": "vl+ocr", "confidence": 1, "verified": true, "flags": [] }
  ],
  "cross_validation": {
    "agreement_rate": 1,
    "unverified_text": [],
    "unverified_count": 0,
    "ocr_only": [],
    "ocr_only_count": 0,
    "ocr_word_count": 7
  },
  "measurements": null,
  "abstained": [],
  "crop_applied": null,
  "provenance": {
    "model": "llava:13b",
    "options": { "temperature": 0, "seed": 42, "num_predict": 2048 },
    "request": { "format": "json-schema", "keep_alive": null },
    "image": { "input_dimensions": { "width": 1200, "height": 760 },
               "sent_dimensions": { "width": 1200, "height": 760 },
               "downscaled": false, "coordinate_space": "sent_dimensions" },
    "metrics": { "time_to_first_token_ms": 210, "total_ms": 7400, "likely_model_load": false },
    "warnings": []
  }
}
```

**Text classification (`source`)**:

| `source` | Meaning | Confidence |
|----------|---------|-----------|
| `vl+ocr` | Vision model and OCR agree | raised (max 1) |
| `vl` | Only the model saw it, but the box is plausible and in-crop | halved, flagged `single_source_unverified` |
| `unverified` | No OCR support **and** no plausible box | quartered, flagged `no_ocr_support` + `no_plausible_box` |
| `ocr` | Tesseract read it but the model did not report it | OCR confidence; flagged `ocr_only_not_reported_by_model` |

`text_items[]` includes OCR-sourced entries, so iterating it yields the text the
server actually believes is present rather than only what the model volunteered.
`cross_validation.text_items_count` and `text_items_sources` let a consumer check
that the two views agree.

`unreadable[]` entries carry `source: "model"` and `verified: false`. An entry is
marked `contested` when OCR read corresponding text — `contest_type: "precise"`
(the named text was read) or `"blanket"` (a generic "too small to read" claim
while OCR read strings in the same image) — with the OCR evidence in
`contested_by` and a warning raised.

**Quantitative requests** return `answered_by: "deterministic-measurement"` and
`model_consulted: false`, with numbers taken from the `measurements` block. The
model is not consulted, so the answer remains correct when Ollama is down.

**Failure to parse** is reported, not hidden: if the model returns prose,
`parsed` is `false`, `claims` is empty, a warning explains the refusal, and the
raw text is retained under `raw_model_output`.

**Large responses stay valid JSON.** Responses are capped at
`MAX_RESPONSE_TEXT_CHARS` (default 12000). Rather than slicing the text mid-token
(which produced unparseable output), the server shortens the largest string
fields and then trims the longest arrays, always returning a parseable document.
A `_truncation` block records exactly what happened:

```json
"_truncation": {
  "truncated": true,
  "max_chars": 12000,
  "actions": [
    { "action": "shortened_string", "path": "$.summary", "original_chars": 260 },
    { "action": "dropped_array_items", "path": "$.text_items", "dropped": 3, "remaining": 9, "was": 12 }
  ],
  "note": "... treat trimmed lists as incomplete."
}
```

Treat any list named in `actions` as **incomplete** — `remaining` tells you how
many items were kept. Raise `MAX_RESPONSE_TEXT_CHARS` to avoid trimming
altogether.

---

## Accuracy and Provenance

This server is deliberately conservative about what it will assert. See
[`ACCURACY.md`](ACCURACY.md) for the full design and evidence.

**Principles**

1. **Measure, don't guess.** Anything with a numeric answer (contrast, counts,
   sizes, colours) is computed from pixels. Asking a vision model for a
   measurement is not supported.
2. **Constrain the output.** Structured calls send Ollama a JSON schema via
   `format`, with `temperature: 0` and an explicit `seed`.
3. **Cross-validate.** Model-read text is checked against Tesseract OCR.
   Unsupported text is flagged rather than reported as fact.
4. **Encourage abstention.** The schema requires an `abstained[]` field, and the
   prompt states a missing answer is better than a wrong one.
5. **Always report provenance.** Every response says which model answered, with
   what sampling options, at what resolution, and whether anything was
   downscaled.

**Provenance footer on legacy tools**

The pre-existing text tools (`analyze_image`, `find_text_element`,
`compare_images`, `browser_screenshot_analysis`, `detect_ui_elements`) append a
footer to their text output:

```
[provenance] model=llava:13b temperature=0 seed=42 ttft_ms=210 total_ms=7400 sent=1200x760 downscaled=false
[warnings] ...
```

**Honest timeouts**

Vision requests are streamed so the server can tell *loading a model* from
*inferring*. A timeout error reports the model, elapsed time, first-token time,
which models currently hold memory (`/api/ps`), a diagnosis, and the vision
models **actually installed on this host**. It no longer suggests a model that
may not be installed.

**Model residency in health checks**

`check_vision_health` reports which models are resident right now and how much
VRAM each holds, plus installed vision models — the context needed to explain a
slow first call.

**Environment variables for accuracy tuning**

| Variable | Default | Description |
|----------|---------|-------------|
| `VISION_TEMPERATURE` | `0` | Sampling temperature for vision calls. |
| `VISION_SEED` | `42` | Sampling seed for reproducibility. |
| `VISION_NUM_PREDICT` | `2048` | Cap on generated tokens per call. |
| `OLLAMA_KEEP_ALIVE` | *(Ollama default, 5m)* | How long a model stays resident. `0` unloads after each call. |

---

## Image Input Formats

All tools accept image input through a resolution pipeline:

1. **Base64 Data URI**: `data:image/png;base64,<BASE64_STRING>`
2. **`file://` URI**: Converted to a local path on the server host.
3. **Home Directory Expansion**: `~/path` is expanded to the server's home directory.
4. **HTTP(S) URL**: Fetched, validated, and saved to `DOWNLOAD_DIR` before processing.
5. **Local Filesystem Path**: Resolved only if the file exists on the **server host** (not the client).
6. **Pure Base64 Fallback**: If the string length exceeds 50 characters, it is decoded and validated as an image.
7. **Upload Reference**: `upload://<filename>` — resolved from the server's `UPLOAD_DIR`.
8. **Download Reference**: `download://<filename>` — resolved from the server's `DOWNLOAD_DIR` (returned by the `download_image` tool).

> **Remote Host Rule**: The MCP Vision Server runs on a remote host. Local filesystem paths (e.g. `/tmp/...`, `C:\...`, `~/...`) are not visible to the server unless they exist on the server host. Transmit images as **Base64 Data URIs**, **HTTP URLs**, or **upload:// references**.

Supported formats: PNG, JPEG, GIF, WEBP, BMP.

All images are normalized to PNG via Sharp before OCR or AI analysis.

## Binary Response Streaming

To reduce MCP response payloads, tools that generate images save their output to `FEEDBACK_DIR` and include a file path in the JSON response text. The Base64 image is still included in the `image` content block for backward compatibility, but clients can switch to reading the file directly for large outputs.

| Tool | Output File Field | Description |
|------|------------------|-------------|
| `preprocess_and_crop` | `output_file_path` | Preprocessed PNG |
| `browser_screenshot_annotation` | `output_file_path` | Annotated PNG |
| `detect_ui_elements` | `overlay_file_path` | Overlay PNG (when `return_overlay=true`) |
| `visual_diff` | `diff_file_path` | Diff PNG |
| `textual_visual_feedback` | `screenshot.file_path` | Screenshot PNG |

## Testing

The project includes test scripts to validate end-to-end tool execution and edge-case handling.

### Quick Transport Test (`test-mcp.js`)

```bash
node test-mcp.js
```

Tests both Streamable HTTP (`/mcp`) and legacy SSE (`/sse`) transports with basic happy-path tool calls.

### Comprehensive Edge-Case Test Runner (`test_runner.mjs`)

```bash
node test_runner.mjs
```

Generates synthetic images in memory using Sharp and runs a full matrix of happy-path and edge-case tests:

- `check_vision_health`: Ensures Ollama, Tesseract, and Sharp report status cleanly.
- `fast_ocr_tesseract`: Happy path with valid Base64; edge case with invalid/truncated Base64.
- `preprocess_and_crop`: Happy path with valid crop + grayscale; edge case with out-of-bounds crop.
- `analyze_image`: Happy path with prompt; edge case with empty prompt (default fallback).
- `find_text_element`: Happy path with valid query; edge case with missing query.
- `compare_images`: Happy path with 2+ images; edge case with single image (minimum length check).
- `browser_screenshot_analysis`: Happy path with focus/detail variants; happy path with default parameters.
- `browser_screenshot_annotation`: Happy path with label + box; edge case with empty annotations array and invalid annotation item; arrow + circle with `return_base64=false`.
- `visual_diff`: Happy path with two images and custom threshold/color; edge case with single image (minimum length check).
- `detect_ui_elements`: Happy path with no overlay; edge case with missing image_source.
- `textual_visual_feedback`: Happy path with OCR; happy path with DOM + CSS; edge case with missing image_source.
- `extract_semantic_page`: Happy path with comprehensive HTML; edge case with empty HTML.
- `generate_repo_graph`: Happy path on current repo; edge case with invalid path.

### Upload Endpoint Test

```bash
# Upload a test image
curl -X POST http://localhost:11402/upload \
  --data-binary @test-image.png \
  -H "Content-Type: image/png"

# Expected response
# {"uploadRef":"upload://<uuid>.png","filename":"<uuid>.png","mimeType":"image/png","size":5242880,"width":1920,"height":1080}
```

## Error Handling

All tool errors are caught and returned as structured MCP error responses. The server returns structured JSON inside `content[0].text` for machine-readable errors:

```json
{
  "content": [
    {
      "type": "text",
      "text": "{ \"error\": true, \"code\": \"INVALID_ANNOTATION\", \"message\": \"Each annotation must be an object with a 'type' field.\", \"validTypes\": [\"label\",\"box\",\"arrow\",\"circle\"], \"suggestions\": [\"Ensure 'label' annotations include 'text'\", \"Ensure coordinate fields (x, y) are numbers\"] }"
    }
  ],
  "isError": true
}
```

Agent clients can parse the JSON to auto-recover or display actionable suggestions. Simple tool execution errors fall back to the original text response with `isError: true`.

Process-level guards (`uncaughtException`, `unhandledRejection`) are installed at server startup to prevent the Node.js process from crashing unexpectedly.

## Utility Scripts

| File | Description |
|------|-------------|
| `upload-helper.js` | Client-side library automating the upload-then-reference flow. |
| `client-example.js` | Runnable examples demonstrating `upload-helper.js` usage. |
| `cleanupUploads()` | Runs every 5 minutes to delete files in `UPLOAD_DIR` older than 15 minutes. |
| `cleanupFeedback()` | Runs every 5 minutes to delete files in `FEEDBACK_DIR` older than 15 minutes. |
| `/health` | Simple Express health check returning JSON status and timestamp. |

## License

ISC