Hybrid Vision MCP Server

A Model Context Protocol (MCP) server that exposes vision capabilities over HTTP, SSE, and Streamable HTTP transports. It bridges local vision engines—Tesseract.js (WASM OCR) and Sharp (image preprocessing)—with local Ollama vision models for image analysis, comparison, text localization, and browser screenshot analysis/annotation.

Features

• MCP Standard Compliance: Exposes tools via @modelcontextprotocol/sdk v1.29.0 using Streamable HTTP and SSE transports.

• Multi-Transport Support: Works with /mcp (streamable HTTP), /sse (legacy SSE), and /messages endpoints.

• Local-First Vision Stack:

  • OCR: Fast CPU-based Tesseract.js WebAssembly engine (shipped with eng.traineddata).

  • Preprocessing: Sharp-powered crop, grayscale, and sharpen filters with boundary validation; returns full Base64 PNG via proper MCP image content blocks.

  • AI Analysis: Local Ollama vision models for description, comparison, element localization, rich browser screenshot analysis, visual diff, UI element detection, textual visual feedback generation, and semantic page extraction.

  • Annotation Engine: SVG-based overlay system for rendering labels, bounding boxes, arrows, and circles on images, returned as annotated PNGs.

  • Repository Analysis: Structural repo mapping via Graphviz DOT and JSON outputs for deep codebase understanding.

• Accuracy Hardening: Deterministic measurement (measure_image), schema-constrained and cross-validated analysis (analyze_image_structured), WCAG contrast maths, OCR cross-validation of model-read text, and full provenance on every response. See ACCURACY.md.

• Flexible Image Input: Accepts Base64 Data URIs, HTTP(S) URLs, file:// URIs, local filesystem paths, upload:// references, and direct file uploads via /upload.

• Structured Responses: Tools return JSON metadata in text blocks and full image data in image content blocks per MCP spec. Large output images are saved to disk and referenced by path to reduce payload bloat.

• Hardened Upload Endpoint: Binary image upload via /upload with MIME validation, magic-number verification, rate limiting, CORS support, and automatic cleanup of stale files.

• Client Automation: upload-helper.js automates the upload-then-reference flow for programmatic MCP clients.

Related MCP server: ollama-vision-mcp

Prerequisites

• Node.js >= 18.x (ESM required)

• Ollama running locally (or reachable via OLLAMA_HOST)

• At least one vision model pulled in Ollama, e.g.:

  ollama pull llava:13b
  ollama pull qwen3-vl:30b

Installation

git clone <repository-url>
cd hybrid-vision-mcp
npm install

Environment Variables

The server loads configuration from environment variables. You can set them in three ways (listed in precedence order, highest first):

1. Shell environment — exported before node index.js

2. .env file — a file named .env in the project root is loaded automatically at startup (values only apply if not already set in the shell)

3. Code defaults — hardcoded fallbacks in index.js

Copy .env.example to .env and edit:

cp .env.example .env

Running the Server

# With .env file (recommended)
cp .env.example .env
# edit .env to suit your setup, then:
node index.js

# Or with inline environment variables:
PORT=11402 VISION_MODEL_FAST="llava:13b" VISION_MODEL_HEAVY="qwen3-vl:30b" node index.js

The server listens on 0.0.0.0 and exposes:

• Health: GET http://localhost:<PORT>/health

• Streamable HTTP (MCP): POST http://localhost:<PORT>/mcp

• Legacy SSE: GET http://localhost:<PORT>/sse + POST http://localhost:<PORT>/messages

• Image Upload: POST http://localhost:<PORT>/upload (binary body, returns upload://<filename> reference)

If the default port 11402 is occupied by another service, set a custom port via the PORT environment variable:

PORT=3001 node index.js

Then point your MCP client configuration to the same port.

Direct Image Upload Workflow

To avoid the ~33% size overhead and token cost of Base64 encoding, upload images directly to the /upload endpoint and pass the returned upload:// reference to any tool.

Step 1: Upload the image

curl -X POST http://localhost:11402/upload \
  --data-binary @screenshot.png \
  -H "Content-Type: image/png"

Response:

{
  "uploadRef": "upload://<uuid>.png",
  "filename": "<uuid>.png",
  "mimeType": "image/png",
  "size": 5242880,
  "width": 1920,
  "height": 1080
}

Step 2: Pass the reference to a tool

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "analyze_image",
    "arguments": {
      "image_source": "upload://<uuid>.png",
      "prompt": "Describe this screenshot."
    }
  }
}

Step 3: Retrieve output images from disk

Tools that produce large output images (preprocess_and_crop, browser_screenshot_annotation, detect_ui_elements, visual_diff) save results to FEEDBACK_DIR and include an output_file_path (or diff_file_path, overlay_file_path) in the JSON response. Clients can read the binary directly instead of parsing Base64 from the MCP response.

Upload Rules

• Max size: 20 MB (configurable via MAX_UPLOAD_SIZE_MB).

• Allowed types: PNG, JPEG, WEBP, GIF, BMP.

• Validation: Magic-number verification ensures the file content matches the declared MIME type.

• Rate limit: 10 uploads per minute per IP (configurable via UPLOAD_RATE_LIMIT).

• Cleanup: Uploads are automatically deleted after 15 minutes. Feedback files follow the same TTL.

MCP SDK Request Timeout

Heavy vision tools (analyze_image, browser_screenshot_analysis, detect_ui_elements, find_text_element, compare_images) rely on local Ollama vision models that may take longer than the MCP SDK's default 60-second request timeout to process large images.

Symptom: MCP error -32001: Request timed out when calling heavy vision tools.

Solution: Pass a timeout option to client.callTool() that matches or exceeds OLLAMA_TIMEOUT_MS:

const result = await client.callTool(
  { name: "analyze_image", arguments: { image_source: url, prompt: "..." } },
  undefined,
  { timeout: 300000 }  // 5 minutes
);

The upload-helper.js library accepts an optional requestTimeout parameter in its constructor. When set, it automatically passes the timeout to every callTool and callToolConnected invocation:

const helper = new MCPUploadHelper("http://localhost:11402", 300000);

The MCP_REQUEST_TIMEOUT_MS environment variable can be used to configure this value. See the Environment Variables table above.

Ollama Request Queuing

All tools that call Ollama (analyze_image, find_text_element, compare_images, browser_screenshot_analysis, detect_ui_elements, visual_diff) share a single in-process concurrency gate. Vision models are GPU/unified-memory bound, and running several inference calls in parallel has been observed to crash the host — so by default (MAX_PARALLEL_OLLAMA_REQUESTS=1) only one Ollama request runs at a time; additional calls wait in an in-process queue rather than being sent to Ollama concurrently.

• Increase MAX_PARALLEL_OLLAMA_REQUESTS only if the host has confirmed headroom to run more than one vision model inference simultaneously.

• MAX_OLLAMA_QUEUE_SIZE (default 20) caps how many callers may wait for a slot; once exceeded, new requests fail fast with a clear error instead of piling up indefinitely.

• Waiting for a queue slot happens before the per-request Ollama timeout clock starts, so a busy queue doesn't erode a caller's OLLAMA_TIMEOUT_MS budget — but a client-side MCP request timeout (MCP_REQUEST_TIMEOUT_MS / timeout in callTool()) still applies across the whole wait-plus-inference duration.

• check_vision_health reports current queue depth (Ollama Concurrency: X/Y active, Z queued).

File Input Schema Annotations

All image-accepting tools now declare their input fields with x-mcp-file annotations in the tool schema. This signals to MCP clients that the field accepts file uploads, enabling native file-picker UI where supported.

Example schema excerpt for analyze_image:

{
  "name": "analyze_image",
  "inputSchema": {
    "type": "object",
    "properties": {
      "image_source": {
        "type": "string",
        "description": "Image to analyze. Accepts: Base64 Data URI, HTTP URL, local file path, or upload://<filename>.",
        "x-mcp-file": true,
        "x-mcp-file-accept": ["image/png", "image/jpeg", "image/webp", "image/gif"]
      }
    },
    "required": ["image_source"]
  }
}

Tools annotated with x-mcp-file:

• fast_ocr_tesseract

• preprocess_and_crop

• analyze_image

• find_text_element

• compare_images

• browser_screenshot_analysis

• browser_screenshot_annotation

• detect_ui_elements

• visual_diff

• textual_visual_feedback

Note: x-mcp-file is a non-standard extension aligned with the MCP File Uploads Working Group's SEP-2356 direction. Existing clients ignore unknown schema properties, so this is fully backward-compatible.

Client Helper Library

upload-helper.js automates the upload-then-reference flow for Node.js MCP clients. It detects File, Blob, ArrayBuffer, Buffer, or string inputs; uploads binary data to /upload; and replaces the argument with an upload:// reference before calling the tool.

Usage

import { MCPUploadHelper } from "./upload-helper.js";

const helper = new MCPUploadHelper("http://localhost:11402");

// One-shot tool call with automatic upload
const result = await helper.callTool("analyze_image", {
  image_source: "/path/to/local/screenshot.png",
  prompt: "Describe this screenshot.",
});
console.log(result.content[0].text);

// Batch upload for multi-image tools
const comparison = await helper.callTool("compare_images", {
  image_sources: ["/tmp/before.png", "/tmp/after.png"],
  prompt: "List all visual differences.",
});

// Reusable connected session
const client = await helper.connect();
await helper.callToolConnected("fast_ocr_tesseract", {
  image_source: "data:image/png;base64,...",
  language: "eng",
});
await helper.close();

See client-example.js for runnable demonstrations.

Client Configuration

For MCP clients that use a config.json format (e.g., VS Code or Kilo MCP extensions), add:

{
  "mcpServers": {
    "hybrid-vision": {
      "url": "http://localhost:11402/sse"
    }
  }
}

Available Tools

1. check_vision_health

Check connectivity to the local Ollama service and verify installed vision engines.

Returns: Text report with Ollama connection status, installed models, and engine readiness.

2. fast_ocr_tesseract

Fast CPU-based WebAssembly OCR extraction for text in images.

Returns: Extracted text with confidence score.

3. preprocess_and_crop

Computer vision image preprocessing using Sharp.

Returns: JSON metadata with output_file_path pointing to the saved PNG, plus an image content block with Base64 data.

Crop Validation:

• left and top must be >= 0.

• width and height must be > 0.

• Crop region must be fully contained within the image boundaries.

4. analyze_image

Analyze image contents or extract context using local Ollama vision models.

Returns: Raw text response from the Ollama vision model.

5. find_text_element

Locate specific text, UI elements, or objects visually within an image.

Returns: Text description of the bounding-box coordinates or visual position of the target element.

6. compare_images

Compare two or more images side-by-side using local Ollama vision models.

CRITICAL: image_sources MUST be a raw JSON array of strings. Do NOT pass a single stringified array.

Returns: Raw text response from the Ollama vision model comparing all provided images.

7. browser_screenshot_analysis

Perform high-level visual and semantic analysis of a browser screenshot or UI image. Generates a rich description of layout, components, visual hierarchy, colors, typography, spacing, and overall design "vibe" to help agents understand the current UI state without manual inspection.

Returns: Structured text summary prefixed with analysis metadata.

8. browser_screenshot_annotation

Annotate a screenshot or image with text labels, bounding boxes, arrows, or circles to highlight specific UI components, regions of interest, or action targets.

Annotation Object Schema:

Returns (default return_base64=true):

{
  "content": [
    {
      "type": "text",
      "text": "{ \"success\": true, \"width\": 800, \"height\": 600, \"format\": \"image/png\", \"output_file_path\": \"/tmp/hvm-feedback/...\", \"data_uri_length\": 123456, \"data_uri\": \"data:image/png;base64,...\" }"
    },
    { "type": "image", "data": "<base64_png>", "mimeType": "image/png" }
  ]
}

9. visual_diff

Compare two screenshots and highlight visual changes between them. Generates a pixel-level diff image that colors changed regions, and optionally an AI-generated description of what differs between the before and after states.

Returns: JSON metadata with diff_file_path pointing to the saved PNG, plus an image content block with Base64 data.

10. detect_ui_elements

Detect UI components and interactive elements in a screenshot using a vision model. Returns structured element descriptions with approximate bounding boxes and an optional overlay visualization.

Returns (default return_overlay=false):

{
  "content": [
    {
      "type": "text",
      "text": "{ \"success\": true, \"width\": 800, \"height\": 600, \"detection\": \"Buttons: [x=...], Inputs: [x=...]\" }"
    }
  ]
}

Returns (when return_overlay=true):

{
  "content": [
    {
      "type": "text",
      "text": "{ \"success\": true, \"width\": 800, \"height\": 600, \"detection\": \"...\", \"overlay_data_uri\": \"data:image/png;base64,...\", \"overlay_file_path\": \"/tmp/hvm-feedback/...\" }"
    },
    { "type": "image", "data": "<base64_png>", "mimeType": "image/png" }
  ]
}

11. textual_visual_feedback

Generate a concise feedback object in JSON format integrating a screenshot, DOM tree, CSS styles, and OCR-derived text data. The screenshot is saved to a local file to avoid context window overflow, and only metadata plus the file path are returned.

Returns:

{
  "content": [
    {
      "type": "text",
      "text": "{ \"success\": true, \"timestamp\": \"2024-...\", \"screenshot\": { \"mime_type\": \"image/png\", \"width\": 800, \"height\": 600, \"data_uri_length\": 954504, \"file_path\": \"/tmp/hvm-feedback/1700000000000-abc.png\" }, \"ocr\": { \"enabled\": true, \"confidence\": 85, \"text\": \"...\" }, \"dom\": { \"provided\": true, \"fragment_length\": 42 }, \"css\": { \"provided\": false } }"
    }
  ]
}

12. extract_semantic_page

Extract structured semantic layout from HTML using DomDistiller-inspired algorithms. Produces a structured Control Map with headings, navigation, content blocks, forms, tables, and other semantic elements.

Returns:

{
  "content": [
    {
      "type": "text",
      "text": "{ \"success\": true, \"control_map\": { \"title\": \"...\", \"headings\": [...], \"navigation\": {...}, \"main_content\": [...], \"lists\": [...], \"forms\": [...], \"tables\": [...], \"media\": [...], \"footer\": {...} } }"
    }
  ]
}

13. generate_repo_graph

Generate repository structural map using Graphviz DOT and JSON formats. Analyzes file tree, classifies files by extension, builds dependency-like parent-child graph for deep codebase understanding.

Returns:

{
  "content": [
    {
      "type": "text",
      "text": "{ \"success\": true, \"graph\": { \"node_count\": 42, \"edge_count\": 38, \"nodes\": [...], \"edges\": [...] } }"
    },
    {
      "type": "text",
      "text": "--- Graphviz DOT ---\ndigraph repo {\n  rankdir=TB;\n  ...\n}"
    }
  ]
}

14. download_image

Download an image from a URL into the server's DOWNLOAD_DIR and return a download:// reference that can be passed to any other tool. Validates the URL, checks content-type, verifies image magic bytes, and enforces size limits.

Returns:

{
  "content": [
    {
      "type": "text",
      "text": "{ \"success\": true, \"downloadRef\": \"download://<uuid>.png\", \"file_path\": \"/abs/path/to/tmp/hvm-downloads/<uuid>.png\", \"original_url\": \"https://...\", \"mime_type\": \"image/png\", \"size\": 123456, \"width\": 800, \"height\": 600 }"
    }
  ]
}

Error scenarios:

• Invalid URL format (not http/https)

• DNS/network failure

• HTTP error (4xx/5xx)

• Wrong content-type (e.g., text/html)

• Payload does not contain valid image magic bytes

• Empty response

• File exceeds MAX_DOWNLOAD_SIZE_MB

---

### 15. `measure_image`

Answer **measurable** questions with code instead of a language model. WCAG
contrast, dominant/background colours, box counts, and region content are
computed from pixels using Sharp, so results are reproducible and available
even when Ollama is unavailable or contended. **No vision model is consulted.**

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `image_source` | string | **Yes** | — | Base64 Data URI, HTTP URL, local path, or `upload://<filename>`. |
| `mode` | string | No | `"all"` | `contrast`, `colors`, `boxes`, `layout`, or `all`. |
| `region` | object | No | — | Region of interest `{ left, top, width, height }`. **Required for meaningful contrast.** |
| `box_color` | string | No | — | Border colour (hex) of the boxes to count. Required for `boxes`. |
| `background_color` | string | No | — | Explicit background colour. If omitted, the region's modal colour is used. |
| `tolerance` | number | No | `24` | RGB distance tolerance for colour matching (0-255). |
| `large_text` | boolean | No | `false` | Assess against WCAG large-text thresholds (3:1). |
| `ink_threshold` | number | No | `4` | RGB distance from the background above which a pixel counts as ink. Lower it to catch text closer to the background. |
| `background_mode` | string | No | `"global"` | `"global"` = one modal background colour (correct for flat UI screenshots). `"local"` = per-tile robust background with a noise-scaled threshold, for photographic/gradient/multi-tone images. |
| `tile_size` | number | No | `48` | Tile edge in pixels for `background_mode: "local"`. |
| `noise_factor` | number | No | `4` | `k` in `threshold = max(ink_threshold, k * local_noise_sigma)` for local mode. |
| `include_decorative` | boolean | No | `false` | Include decorative (thin hollow rectangle) colours in the summary instead of reporting them separately. |
| `decorative_colors` | array[string] | No | — | Hex colours to treat as decorative explicitly. |

**Contrast enumerates every text colour and reports the worst case.** The old
behaviour selected only the most legible colour, which meant an image containing
large text at 2.14:1 and 1.04:1 could be summarised as `wcag_aa: true` — reading
as "no contrast problems". Now:
```json
"contrast": {
  "worst":  { "foreground": "#1e1c18", "contrast_ratio": 1.04, "wcag_aa": false },
  "best":   { "foreground": "#e8dfd0", "contrast_ratio": 13.42, "wcag_aa": true },
  "failing_count": 2,
  "passing_count": 3,
  "evaluated_count": 5,
  "all_meet_aa": false,
  "wcag_aa": false,
  "contrast_ratio": 1.04,
  "colours": [
    { "foreground": "#1e1c18", "pixel_count": 2241, "contrast_ratio": 1.04, "wcag_aa": false, "meets_aa": false },
    { "foreground": "#484f58", "pixel_count": 2792, "contrast_ratio": 2.14, "wcag_aa": false, "meets_aa": false },
    { "foreground": "#a09588", "pixel_count": 639,  "contrast_ratio": 6.03, "wcag_aa": true },
    { "foreground": "#7daa7a", "pixel_count": 1052, "contrast_ratio": 6.67, "wcag_aa": true },
    { "foreground": "#e8dfd0", "pixel_count": 4539, "contrast_ratio": 13.42, "wcag_aa": true }
  ],
  "excluded": [ { "foreground": "#4a433c", "contrast_ratio": 1.82, "reason": "decorative: thin hollow rectangle geometry (border/rule), not text" } ],
  "notes": ["...", "2 of 5 evaluated text colour(s) fail WCAG AA: ..."]
}

Semantics change: wcag_aa at the contrast level now aliases the worst case (all_meet_aa) — true only when every evaluated text colour passes. best.contrast_ratio preserves the previous number, and foreground / contrast_ratio now refer to the worst case. See ACCURACY.md §5b.

Nothing is silently dropped. Excluded (decorative) and skipped (sub-threshold) colours are listed with their ratio and pixel count and described in notes, and they are excluded from all_meet_aa. If no text colour is assessable the result is measurable: false with an abstained[] entry rather than a verdict. In background_mode: "local", clusters judged to be residual noise rather than text appear under suspected_noise (also excluded from the verdict, also disclosed).

Photographic and gradient images. A single global background colour only models flat UI screenshots well. When it explains less than half the region, the result carries a background_fit block and a warning in notes:

"background_fit": {
  "background": "#90877d",
  "explained_fraction": 0.007,
  "tolerance": 8,
  "adequate": false,
  "applicable": true,
  "warnings": ["A single global background colour accounts for only 0.7% ..."]
}

A fit between 0.5 and 0.8 is marginal: it clears the adequacy floor but is still a weak model, so it is disclosed too, and a verdict is never silent about it.

The adequate flag is only meaningful when a single background applies. In a multi-plateau region it is null (with applicable: false and a reason), so it cannot read as "fine" for a value that does not apply there.

For such images pass background_mode: "local", which uses a per-tile median background with a threshold that scales with each tile's measured noise. On a text-free photo that yields zero reported colours (rather than the whole gradient counting as one), and it recovers text tones a global background misses. The mode is opt-in because a tile straddling two flat panels can over-threshold — measured, not assumed. See ACCURACY.md §5c.

Multi-panel UIs (sidebar + content). A single background colour cannot model a UI with two or more large flat panels: the non-modal panel used to become one enormous "ink" blob whose colour was the panel fill, and text sitting on that panel disappeared entirely — so a 1.64:1 sidebar string could be reported as all_meet_aa: true. When two or more large flat plateaus are detected, every plateau is treated as background and each text colour is measured against the fill it actually sits on:

"background_model": "multi-plateau",
"plateaus": [ { "hex": "#d2d2d2", "share": 0.6928 }, { "hex": "#161616", "share": 0.2892 } ],
"colours": [
  { "foreground": "#3c3c3c", "contrast_ratio": 1.64, "wcag_aa": false,
    "measured_against": "#161616", "measured_against_plateau": "#161616", "local_background": "#161616" },
  { "foreground": "#282828", "contrast_ratio": 9.75, "wcag_aa": true,
    "measured_against": "#d2d2d2", "measured_against_plateau": "#d2d2d2" }
],
"panel_fills": [ { "foreground": "#d2d2d2", "reason": "panel fill: a large near-solid region ..." } ],
"notes": [ "Multi-plateau region: 2 large flat colour plates were detected ..." ]

This is structural, so it fires even when the modal colour explains most of the area (a 30/70 layout scores explained_fraction: 0.693, which a single-colour adequacy test would call fine). A plateau must pass four measured tests — minimum region size, component dominance (a panel is ONE blob; large text is many), flatness (a panel is one exact colour; a gradient band is not) and connectivity. With fewer than two plateaus the result is unchanged. See ACCURACY.md §5e.

Tiled layouts are panels too. A dashboard repeats its cards, so a card fill is "many small identical blobs" — the same signature as a glyph run. A second, structural plateau path recognises a repeated fill by solidity and size per blob rather than blob count (>= 2 blobs, each >= 0.4% of the region, fill >= 0.85, similar sizes). Without it the page background around the cards became one huge component and was reported as failing text on an image where every colour passes. plateaus[].detection says which path matched ("dominant-blob" or "tiled").

"plateaus": [
  { "hex": "#1a1814", "share": 0.3768, "detection": "dominant-blob" },
  { "hex": "#2d2822", "share": 0.6021, "detection": "tiled", "solid_component_count": 12 }
],
"background_model": "multi-plateau",
"colours": [ { "foreground": "#e8dfd0", "contrast_ratio": 11.05, "wcag_aa": true } ]

A background region is never a failing text colour. A near-background cluster whose blobs are individually large is a region separated by panels, not a text run, so it is listed under background_regions[] and excluded from the verdict (with a note). Real text averages <= 0.10% of the region per blob against 37.7% for a page background, so this cannot drop a text run. See ACCURACY.md §5g.

A text colour is never a plateau. A plateau blob must be either the outermost colour (the background, with panels cut out of it, so a low fill is expected) or an inset blob that is not a glyph RING. A ring is recognised by one large enclosed aperture; a content-dense panel (a chart card, a table, a thumbnail grid) is perforated by many small holes and is still a panel. Two huge identical glyphs can no longer be mistaken for a background — which had made failing text disappear. See ACCURACY.md §5h-§5i.

Mask reconciliation. When a plateau mask removed a colour, the region is re-enumerated with no mask. Any failing colour the masked result dropped is disclosed under mask_reconciliation and in notes, with the raw evidence:

"mask_reconciliation": {
  "unmasked_failing_colours": [ {
    "foreground": "#464646", "contrast_ratio": 1.88,
    "component_count": 164, "mean_component_area": 270,
    "detected_plateau": true, "plateau_share": 0.0489, "largest_component_share": 0.0174
  } ],
  "note": "masking a background region also removed colour(s) that an un-masked pass measures as failing. These are usually backgrounds or decoration, but they could be text. The tool does NOT classify which: judge from the reported evidence ..."
}

The gate is UNCONDITIONAL. Every failing colour the mask removed is disclosed — there is no size threshold. The tool does not classify whether a flagged colour is decoration or text; it reports raw evidence so you can judge:

• detected_plateau — whether the colour was itself read as a background plateau (evidence, not a classifier: real text is often read as a plateau too);

• plateau_share — the plateau's coverage of the region. Fragmentation-invariant, but it orders large from small, not decoration from text: a dense glyph run can cover MORE than a bar chart (measured 0.1561 text vs 0.0986 bars);

• component_count, mean_component_area, largest_component_share, pixel_count.

It remains disclosure, not refusal, so it cannot turn a correct verdict into a wrong one.

History (three attempts to add a size gate, all reverted). A gate looks like precision but keeps hiding real text:

• Tenth audit (F12): mean >= 0.4% compared a mean to the mask's single-blob floor, hiding a 2%-blob colour with many small companions.

• Eleventh audit (F13): a union added largestShare >= 2% (true by construction for the mask's path-A) but kept the mean clause.

• Twelfth audit (F14): the mean clause was shown to be anti-correlated with the evidence — adding six small body lines to a 300px failing heading took the colour from disclosed (mean 1.78%) to silent (mean 0.03%) while the largest blob was unchanged. More failing text made the warning disappear. The gate is therefore removed, not re-tuned. See ACCURACY.md §5n.

Consequence: mask_reconciliation now fires on decorative dashboards (the ninth-audit F11 case), because the un-masked pass re-reads a tiled bar colour as failing text. That is the deliberate trade — the disclosure is advisory and names its own evidence, whereas the F14 omission was silent. If you want to ignore decoration, filter on detected_plateau / plateau_share rather than expecting it to be suppressed. Be aware (fourteenth audit F16) that neither field classifies decoration vs text — see ACCURACY.md §5p.

No classification is provided (thirteenth audit F15). An earlier version also emitted shape: "panel-shaped" | "text-sized". It was removed: its threshold was mean = total / N again, so it flipped with fragmentation and inverted on a real pair (9 decorative bars and a 164-glyph failing run were both text-sized). The replacement the audit proposed — the largest connected component — was measured and does not separate them either (0.0185 vs 0.0174). A bar chart and a glyph run are the same kind of object, so no scalar is offered in its place. See ACCURACY.md §5o.

Documentation-guard scope (F16–F26). This repository also carries a lint that scans the docs for prose asserting that an exposed field classifies decoration vs text. The lint fails the build when it sees a claim; what we no longer rely on is its completeness. Its measured recall on the fixture set is 18/19 (a regression score over the known escape cases, not an estimate over unseen prose; P4 is the known miss), printed by the test and enforced so it cannot silently drop. The part we trust is the positive assertion: the docs must contain the explicit disclaimer (they do). The acceptance test that matters is the invariance pair (a claim must get the same verdict with or without each waiver mechanism). See ACCURACY.md §5r–§5v.

The guard is FROZEN (twenty-first audit). Following the round-19 recommendation, no further effort goes into extending this lint; later rounds spend their substance on the measurement engine instead. See ACCURACY.md §5w for that round's two engine defects (a text-free gradient reported as failing text; the darker of two colours in outlined text silently absorbed), §5x for the follow-up (F29): the outlined-text pixel floor was applied per component, so small glyphs lost their failing fill as it split across components — it is now applied to the colour's total across the scan; §5y for the next round (F30/F31): the second-ink path had its own 28×-larger floor, which hid failing ink and admitted accumulated card/text anti-aliasing fringes — it now uses the primary path's reporting floor and gates; §5z (F32): a further divergence on the colour-proximity axis let a real fill that lies on the reference→stroke line be discarded as a fringe — the AA test now rejects a blend only when it is small; §5aa (F34): that size qualifier was applied per component, so a real on-line fill split across glyphs was discarded — it is now applied to the colour's on-line total in the deferred pass. In the second-ink path, no gate may use a per-component quantity without stating why the total is wrong. §5ab (F35): the same per-piece shape appeared in the primary path's large-background-region gate (a split decoration was reported as failing text); the gate now requires the total and solidity, keeping the original mean test for the textured-page backstop. A standing count-invariance test (fixed total, varying piece count) now covers this class; its coverage is measured (§5ac) — it catches F32/F34/F35, and the cases it does not cover are stated rather than assumed.

Returns (excerpt):

{
  "success": true,
  "mode": "contrast",
  "dimensions": { "width": 1200, "height": 760, "format": "png" },
  "measurements": {
    "contrast": {
      "background": { "hex": "#1a1814", "rgb": { "r": 26, "g": 24, "b": 20 } },
      "foreground": { "hex": "#484f58", "rgb": { "r": 72, "g": 79, "b": 88 } },
      "contrast_ratio": 2.14,
      "wcag_aa": false,
      "wcag_aaa": false,
      "required_aa": 4.5,
      "measurable": true,
      "method": "pixel-histogram/exact-colour-mode+max-luminance-delta"
    }
  },
  "abstained": [],
  "provenance": { "model": null, "image": { "sent_dimensions": { "width": 1200, "height": 760 }, "downscaled": false } },
  "disclaimer": "Deterministic measurement. Reproducible without a vision model..."
}

Honest abstention: boxes mode without box_color abstains (counting without knowing the border colour would be a guess). contrast mode with no region does not abstain — it measures the whole frame and says so in notes, exactly as mode: "all" does, so the same question always gets the same answer regardless of mode. Pass a region when the question is about specific text.

Noise is not called out as text. A weak cluster that is not shaped like text (fewer than 3 components or a sub-100px mean component area) is returned under suspected_noise and named as suspected noise in notes, rather than being asserted as a failing text colour. It is excluded from worst/failing_count/all_meet_aa but never hidden.

Local mode has no single background. Under background_mode: "local", background is null (there is no single background by construction, and the global modal colour can coincide with a text tone).

Every colour says what it was measured against. Each colours[] entry carries measured_against (the fill the text sits on) and local_background, so a caller never has to infer which background produced a ratio.

Panel fills are not text colours. In a multi-plateau region the large near-solid panel colours are returned under panel_fills[] and excluded from the text enumeration, so a panel's own fill is never reported as a "failing colour".

Straight strokes are chrome, not glyphs. A component that is long and thin in one axis (≥ 60px long, ≤ 4px across) — a rule, divider or card border — is treated as decorative, so a bordered card grid is not reported as failing text.

A clean verdict is never silent about a weak premise. When the global model reports no failing colour, the region is re-measured with the per-tile (local) model. If local finds a failing tone the global model missed — the signature of a background that varies across the region — the result carries a model_disagreement block and a note asking you to re-measure with a region:

"model_disagreement": {
  "global_all_meet_aa": true,
  "local_all_meet_aa": false,
  "local_failing_count": 1,
  "local_failing_colours": [ { "foreground": "#8c8c8c", "contrast_ratio": 3.81, "measured_against": "#3f3f3f" } ],
  "note": "the two background models disagree on this region ... Treat the clean verdict as UNVERIFIED ... Note that local mode is not generally more accurate: it over-reports on dense flat panels, which is why it is not the default."
}

This is a detector, not an endorsement: on dense flat panels local over-reports, so the note explicitly does not claim local is better.

16. analyze_image_structured

Analyze an image and return structured, evidence-backed output instead of free prose. Every statement is a claims[] entry carrying its own bounding box and confidence; transcribed text is cross-validated against Tesseract OCR and flagged as unverified when it has no OCR support and no plausible box (the fabrication signature). Abstention is first-class. Quantitative questions are answered by deterministic measurement in code, never by the model.

Returns (excerpt):

{
  "success": true,
  "parsed": true,
  "summary": "...",
  "claims": [
    { "claim": "...", "kind": "text", "box": { "left": 40, "top": 40, "width": 200, "height": 40 },
      "confidence": 0.8, "source": "vl", "verified": false, "basis": "vision-model-observation" }
  ],
  "unsupported_claims": [],
  "text_items": [
    { "text": "ALPHA-ONE", "source": "vl+ocr", "confidence": 1, "verified": true, "flags": [] }
  ],
  "cross_validation": {
    "agreement_rate": 1,
    "unverified_text": [],
    "unverified_count": 0,
    "ocr_only": [],
    "ocr_only_count": 0,
    "ocr_word_count": 7
  },
  "measurements": null,
  "abstained": [],
  "crop_applied": null,
  "provenance": {
    "model": "llava:13b",
    "options": { "temperature": 0, "seed": 42, "num_predict": 2048 },
    "request": { "format": "json-schema", "keep_alive": null },
    "image": { "input_dimensions": { "width": 1200, "height": 760 },
               "sent_dimensions": { "width": 1200, "height": 760 },
               "downscaled": false, "coordinate_space": "sent_dimensions" },
    "metrics": { "time_to_first_token_ms": 210, "total_ms": 7400, "likely_model_load": false },
    "warnings": []
  }
}

Text classification (source):

text_items[] includes OCR-sourced entries, so iterating it yields the text the server actually believes is present rather than only what the model volunteered. cross_validation.text_items_count and text_items_sources let a consumer check that the two views agree.

unreadable[] entries carry source: "model" and verified: false. An entry is marked contested when OCR read corresponding text — contest_type: "precise" (the named text was read) or "blanket" (a generic "too small to read" claim while OCR read strings in the same image) — with the OCR evidence in contested_by and a warning raised.

Quantitative requests return answered_by: "deterministic-measurement" and model_consulted: false, with numbers taken from the measurements block. The model is not consulted, so the answer remains correct when Ollama is down.

Failure to parse is reported, not hidden: if the model returns prose, parsed is false, claims is empty, a warning explains the refusal, and the raw text is retained under raw_model_output.

Large responses stay valid JSON. Responses are capped at MAX_RESPONSE_TEXT_CHARS (default 12000). Rather than slicing the text mid-token (which produced unparseable output), the server shortens the largest string fields and then trims the longest arrays, always returning a parseable document. A _truncation block records exactly what happened:

"_truncation": {
  "truncated": true,
  "max_chars": 12000,
  "actions": [
    { "action": "shortened_string", "path": "$.summary", "original_chars": 260 },
    { "action": "dropped_array_items", "path": "$.text_items", "dropped": 3, "remaining": 9, "was": 12 }
  ],
  "note": "... treat trimmed lists as incomplete."
}

Treat any list named in actions as incomplete — remaining tells you how many items were kept. Raise MAX_RESPONSE_TEXT_CHARS to avoid trimming altogether.

Accuracy and Provenance

This server is deliberately conservative about what it will assert. See ACCURACY.md for the full design and evidence.

Principles

1. Measure, don't guess. Anything with a numeric answer (contrast, counts, sizes, colours) is computed from pixels. Asking a vision model for a measurement is not supported.

2. Constrain the output. Structured calls send Ollama a JSON schema via format, with temperature: 0 and an explicit seed.

3. Cross-validate. Model-read text is checked against Tesseract OCR. Unsupported text is flagged rather than reported as fact.

4. Encourage abstention. The schema requires an abstained[] field, and the prompt states a missing answer is better than a wrong one.

5. Always report provenance. Every response says which model answered, with what sampling options, at what resolution, and whether anything was downscaled.

Provenance footer on legacy tools

The pre-existing text tools (analyze_image, find_text_element, compare_images, browser_screenshot_analysis, detect_ui_elements) append a footer to their text output:

[provenance] model=llava:13b temperature=0 seed=42 ttft_ms=210 total_ms=7400 sent=1200x760 downscaled=false
[warnings] ...

Honest timeouts

Vision requests are streamed so the server can tell loading a model from inferring. A timeout error reports the model, elapsed time, first-token time, which models currently hold memory (/api/ps), a diagnosis, and the vision models actually installed on this host. It no longer suggests a model that may not be installed.

Model residency in health checks

check_vision_health reports which models are resident right now and how much VRAM each holds, plus installed vision models — the context needed to explain a slow first call.

Environment variables for accuracy tuning

Image Input Formats

All tools accept image input through a resolution pipeline:

1. Base64 Data URI: data:image/png;base64,<BASE64_STRING>

2. file:// URI: Converted to a local path on the server host.

3. Home Directory Expansion: ~/path is expanded to the server's home directory.

4. HTTP(S) URL: Fetched, validated, and saved to DOWNLOAD_DIR before processing.

5. Local Filesystem Path: Resolved only if the file exists on the server host (not the client).

6. Pure Base64 Fallback: If the string length exceeds 50 characters, it is decoded and validated as an image.

7. Upload Reference: upload://<filename> — resolved from the server's UPLOAD_DIR.

8. Download Reference: download://<filename> — resolved from the server's DOWNLOAD_DIR (returned by the download_image tool).

Remote Host Rule: The MCP Vision Server runs on a remote host. Local filesystem paths (e.g. /tmp/..., C:\..., ~/...) are not visible to the server unless they exist on the server host. Transmit images as Base64 Data URIs, HTTP URLs, or upload:// references.

Supported formats: PNG, JPEG, GIF, WEBP, BMP.

All images are normalized to PNG via Sharp before OCR or AI analysis.

Binary Response Streaming

To reduce MCP response payloads, tools that generate images save their output to FEEDBACK_DIR and include a file path in the JSON response text. The Base64 image is still included in the image content block for backward compatibility, but clients can switch to reading the file directly for large outputs.

Testing

The project includes test scripts to validate end-to-end tool execution and edge-case handling.

Quick Transport Test (test-mcp.js)

node test-mcp.js

Tests both Streamable HTTP (/mcp) and legacy SSE (/sse) transports with basic happy-path tool calls.

Comprehensive Edge-Case Test Runner (test_runner.mjs)

node test_runner.mjs

Generates synthetic images in memory using Sharp and runs a full matrix of happy-path and edge-case tests:

• check_vision_health: Ensures Ollama, Tesseract, and Sharp report status cleanly.

• fast_ocr_tesseract: Happy path with valid Base64; edge case with invalid/truncated Base64.

• preprocess_and_crop: Happy path with valid crop + grayscale; edge case with out-of-bounds crop.

• analyze_image: Happy path with prompt; edge case with empty prompt (default fallback).

• find_text_element: Happy path with valid query; edge case with missing query.

• compare_images: Happy path with 2+ images; edge case with single image (minimum length check).

• browser_screenshot_analysis: Happy path with focus/detail variants; happy path with default parameters.

• browser_screenshot_annotation: Happy path with label + box; edge case with empty annotations array and invalid annotation item; arrow + circle with return_base64=false.

• visual_diff: Happy path with two images and custom threshold/color; edge case with single image (minimum length check).

• detect_ui_elements: Happy path with no overlay; edge case with missing image_source.

• textual_visual_feedback: Happy path with OCR; happy path with DOM + CSS; edge case with missing image_source.

• extract_semantic_page: Happy path with comprehensive HTML; edge case with empty HTML.

• generate_repo_graph: Happy path on current repo; edge case with invalid path.

Upload Endpoint Test

# Upload a test image
curl -X POST http://localhost:11402/upload \
  --data-binary @test-image.png \
  -H "Content-Type: image/png"

# Expected response
# {"uploadRef":"upload://<uuid>.png","filename":"<uuid>.png","mimeType":"image/png","size":5242880,"width":1920,"height":1080}

Error Handling

All tool errors are caught and returned as structured MCP error responses. The server returns structured JSON inside content[0].text for machine-readable errors:

{
  "content": [
    {
      "type": "text",
      "text": "{ \"error\": true, \"code\": \"INVALID_ANNOTATION\", \"message\": \"Each annotation must be an object with a 'type' field.\", \"validTypes\": [\"label\",\"box\",\"arrow\",\"circle\"], \"suggestions\": [\"Ensure 'label' annotations include 'text'\", \"Ensure coordinate fields (x, y) are numbers\"] }"
    }
  ],
  "isError": true
}

Agent clients can parse the JSON to auto-recover or display actionable suggestions. Simple tool execution errors fall back to the original text response with isError: true.

Process-level guards (uncaughtException, unhandledRejection) are installed at server startup to prevent the Node.js process from crashing unexpectedly.

Utility Scripts

License

ISC