Hybrid Vision MCP Server

A Model Context Protocol (MCP) server that exposes vision capabilities over HTTP, SSE, and Streamable HTTP transports. It bridges local vision engines—Tesseract.js (WASM OCR) and Sharp (image preprocessing)—with local Ollama vision models for image analysis, comparison, text localization, and browser screenshot analysis/annotation.

Features

• MCP Standard Compliance: Exposes tools via @modelcontextprotocol/sdk v1.29.0 using Streamable HTTP and SSE transports.

• Multi-Transport Support: Works with /mcp (streamable HTTP), /sse (legacy SSE), and /messages endpoints.

• Local-First Vision Stack:

  • OCR: Fast CPU-based Tesseract.js WebAssembly engine (shipped with eng.traineddata).

  • Preprocessing: Sharp-powered crop, grayscale, and sharpen filters with boundary validation; returns full Base64 PNG via proper MCP image content blocks.

  • AI Analysis: Local Ollama vision models for description, comparison, element localization, rich browser screenshot analysis, visual diff, UI element detection, textual visual feedback generation, and semantic page extraction.

  • Annotation Engine: SVG-based overlay system for rendering labels, bounding boxes, arrows, and circles on images, returned as annotated PNGs.

  • Repository Analysis: Structural repo mapping via Graphviz DOT and JSON outputs for deep codebase understanding.

• Flexible Image Input: Accepts Base64 Data URIs, HTTP(S) URLs, file:// URIs, local filesystem paths, upload:// references, and direct file uploads via /upload.

• Structured Responses: Tools return JSON metadata in text blocks and full image data in image content blocks per MCP spec. Large output images are saved to disk and referenced by path to reduce payload bloat.

• Hardened Upload Endpoint: Binary image upload via /upload with MIME validation, magic-number verification, rate limiting, CORS support, and automatic cleanup of stale files.

• Client Automation: upload-helper.js automates the upload-then-reference flow for programmatic MCP clients.

Prerequisites

• Node.js >= 18.x (ESM required)

• Ollama running locally (or reachable via OLLAMA_HOST)

• At least one vision model pulled in Ollama, e.g.:

  ollama pull llava:13b
  ollama pull qwen3-vl:30b

Installation

git clone <repository-url>
cd hybrid-vision-mcp
npm install

Environment Variables

The server loads configuration from environment variables. You can set them in three ways (listed in precedence order, highest first):

1. Shell environment — exported before node index.js

2. .env file — a file named .env in the project root is loaded automatically at startup (values only apply if not already set in the shell)

3. Code defaults — hardcoded fallbacks in index.js

Copy .env.example to .env and edit:

cp .env.example .env

Running the Server

# With .env file (recommended)
cp .env.example .env
# edit .env to suit your setup, then:
node index.js

# Or with inline environment variables:
PORT=11402 VISION_MODEL_FAST="llava:13b" VISION_MODEL_HEAVY="qwen3-vl:30b" node index.js

The server listens on 0.0.0.0 and exposes:

• Health: GET http://localhost:<PORT>/health

• Streamable HTTP (MCP): POST http://localhost:<PORT>/mcp

• Legacy SSE: GET http://localhost:<PORT>/sse + POST http://localhost:<PORT>/messages

• Image Upload: POST http://localhost:<PORT>/upload (binary body, returns upload://<filename> reference)

If the default port 11402 is occupied by another service, set a custom port via the PORT environment variable:

PORT=3001 node index.js

Then point your MCP client configuration to the same port.

Direct Image Upload Workflow

To avoid the ~33% size overhead and token cost of Base64 encoding, upload images directly to the /upload endpoint and pass the returned upload:// reference to any tool.

Step 1: Upload the image

curl -X POST http://localhost:11402/upload \
  --data-binary @screenshot.png \
  -H "Content-Type: image/png"

Response:

{
  "uploadRef": "upload://<uuid>.png",
  "filename": "<uuid>.png",
  "mimeType": "image/png",
  "size": 5242880,
  "width": 1920,
  "height": 1080
}

Step 2: Pass the reference to a tool

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "analyze_image",
    "arguments": {
      "image_source": "upload://<uuid>.png",
      "prompt": "Describe this screenshot."
    }
  }
}

Step 3: Retrieve output images from disk

Tools that produce large output images (preprocess_and_crop, browser_screenshot_annotation, detect_ui_elements, visual_diff) save results to FEEDBACK_DIR and include an output_file_path (or diff_file_path, overlay_file_path) in the JSON response. Clients can read the binary directly instead of parsing Base64 from the MCP response.

Upload Rules

• Max size: 20 MB (configurable via MAX_UPLOAD_SIZE_MB).

• Allowed types: PNG, JPEG, WEBP, GIF, BMP.

• Validation: Magic-number verification ensures the file content matches the declared MIME type.

• Rate limit: 10 uploads per minute per IP (configurable via UPLOAD_RATE_LIMIT).

• Cleanup: Uploads are automatically deleted after 15 minutes. Feedback files follow the same TTL.

MCP SDK Request Timeout

Heavy vision tools (analyze_image, browser_screenshot_analysis, detect_ui_elements, find_text_element, compare_images) rely on local Ollama vision models that may take longer than the MCP SDK's default 60-second request timeout to process large images.

Symptom: MCP error -32001: Request timed out when calling heavy vision tools.

Solution: Pass a timeout option to client.callTool() that matches or exceeds OLLAMA_TIMEOUT_MS:

const result = await client.callTool(
  { name: "analyze_image", arguments: { image_source: url, prompt: "..." } },
  undefined,
  { timeout: 300000 }  // 5 minutes
);

The upload-helper.js library accepts an optional requestTimeout parameter in its constructor. When set, it automatically passes the timeout to every callTool and callToolConnected invocation:

const helper = new MCPUploadHelper("http://localhost:11402", 300000);

The MCP_REQUEST_TIMEOUT_MS environment variable can be used to configure this value. See the Environment Variables table above.

File Input Schema Annotations

All image-accepting tools now declare their input fields with x-mcp-file annotations in the tool schema. This signals to MCP clients that the field accepts file uploads, enabling native file-picker UI where supported.

Example schema excerpt for analyze_image:

{
  "name": "analyze_image",
  "inputSchema": {
    "type": "object",
    "properties": {
      "image_source": {
        "type": "string",
        "description": "Image to analyze. Accepts: Base64 Data URI, HTTP URL, local file path, or upload://<filename>.",
        "x-mcp-file": true,
        "x-mcp-file-accept": ["image/png", "image/jpeg", "image/webp", "image/gif"]
      }
    },
    "required": ["image_source"]
  }
}

Tools annotated with x-mcp-file:

• fast_ocr_tesseract

• preprocess_and_crop

• analyze_image

• find_text_element

• compare_images

• browser_screenshot_analysis

• browser_screenshot_annotation

• detect_ui_elements

• visual_diff

• textual_visual_feedback

Note: x-mcp-file is a non-standard extension aligned with the MCP File Uploads Working Group's SEP-2356 direction. Existing clients ignore unknown schema properties, so this is fully backward-compatible.

Client Helper Library

upload-helper.js automates the upload-then-reference flow for Node.js MCP clients. It detects File, Blob, ArrayBuffer, Buffer, or string inputs; uploads binary data to /upload; and replaces the argument with an upload:// reference before calling the tool.

Usage

import { MCPUploadHelper } from "./upload-helper.js";

const helper = new MCPUploadHelper("http://localhost:11402");

// One-shot tool call with automatic upload
const result = await helper.callTool("analyze_image", {
  image_source: "/path/to/local/screenshot.png",
  prompt: "Describe this screenshot.",
});
console.log(result.content[0].text);

// Batch upload for multi-image tools
const comparison = await helper.callTool("compare_images", {
  image_sources: ["/tmp/before.png", "/tmp/after.png"],
  prompt: "List all visual differences.",
});

// Reusable connected session
const client = await helper.connect();
await helper.callToolConnected("fast_ocr_tesseract", {
  image_source: "data:image/png;base64,...",
  language: "eng",
});
await helper.close();

See client-example.js for runnable demonstrations.

Client Configuration

For MCP clients that use a config.json format (e.g., VS Code or Kilo MCP extensions), add:

{
  "mcpServers": {
    "hybrid-vision": {
      "url": "http://localhost:11402/sse"
    }
  }
}

Available Tools

1. check_vision_health

Check connectivity to the local Ollama service and verify installed vision engines.

Returns: Text report with Ollama connection status, installed models, and engine readiness.

2. fast_ocr_tesseract

Fast CPU-based WebAssembly OCR extraction for text in images.

Returns: Extracted text with confidence score.

3. preprocess_and_crop

Computer vision image preprocessing using Sharp.

Returns: JSON metadata with output_file_path pointing to the saved PNG, plus an image content block with Base64 data.

Crop Validation:

• left and top must be >= 0.

• width and height must be > 0.

• Crop region must be fully contained within the image boundaries.

4. analyze_image

Analyze image contents or extract context using local Ollama vision models.

Returns: Raw text response from the Ollama vision model.

5. find_text_element

Locate specific text, UI elements, or objects visually within an image.

Returns: Text description of the bounding-box coordinates or visual position of the target element.

6. compare_images

Compare two or more images side-by-side using local Ollama vision models.

CRITICAL: image_sources MUST be a raw JSON array of strings. Do NOT pass a single stringified array.

Returns: Raw text response from the Ollama vision model comparing all provided images.

7. browser_screenshot_analysis

Perform high-level visual and semantic analysis of a browser screenshot or UI image. Generates a rich description of layout, components, visual hierarchy, colors, typography, spacing, and overall design "vibe" to help agents understand the current UI state without manual inspection.

Returns: Structured text summary prefixed with analysis metadata.

8. browser_screenshot_annotation

Annotate a screenshot or image with text labels, bounding boxes, arrows, or circles to highlight specific UI components, regions of interest, or action targets.

Annotation Object Schema:

Returns (default return_base64=true):

{
  "content": [
    {
      "type": "text",
      "text": "{ \"success\": true, \"width\": 800, \"height\": 600, \"format\": \"image/png\", \"output_file_path\": \"/tmp/hvm-feedback/...\", \"data_uri_length\": 123456, \"data_uri\": \"data:image/png;base64,...\" }"
    },
    { "type": "image", "data": "<base64_png>", "mimeType": "image/png" }
  ]
}

9. visual_diff

Compare two screenshots and highlight visual changes between them. Generates a pixel-level diff image that colors changed regions, and optionally an AI-generated description of what differs between the before and after states.

Returns: JSON metadata with diff_file_path pointing to the saved PNG, plus an image content block with Base64 data.

10. detect_ui_elements

Detect UI components and interactive elements in a screenshot using a vision model. Returns structured element descriptions with approximate bounding boxes and an optional overlay visualization.

Returns (default return_overlay=false):

{
  "content": [
    {
      "type": "text",
      "text": "{ \"success\": true, \"width\": 800, \"height\": 600, \"detection\": \"Buttons: [x=...], Inputs: [x=...]\" }"
    }
  ]
}

Returns (when return_overlay=true):

{
  "content": [
    {
      "type": "text",
      "text": "{ \"success\": true, \"width\": 800, \"height\": 600, \"detection\": \"...\", \"overlay_data_uri\": \"data:image/png;base64,...\", \"overlay_file_path\": \"/tmp/hvm-feedback/...\" }"
    },
    { "type": "image", "data": "<base64_png>", "mimeType": "image/png" }
  ]
}

11. textual_visual_feedback

Generate a concise feedback object in JSON format integrating a screenshot, DOM tree, CSS styles, and OCR-derived text data. The screenshot is saved to a local file to avoid context window overflow, and only metadata plus the file path are returned.

Returns:

{
  "content": [
    {
      "type": "text",
      "text": "{ \"success\": true, \"timestamp\": \"2024-...\", \"screenshot\": { \"mime_type\": \"image/png\", \"width\": 800, \"height\": 600, \"data_uri_length\": 954504, \"file_path\": \"/tmp/hvm-feedback/1700000000000-abc.png\" }, \"ocr\": { \"enabled\": true, \"confidence\": 85, \"text\": \"...\" }, \"dom\": { \"provided\": true, \"fragment_length\": 42 }, \"css\": { \"provided\": false } }"
    }
  ]
}

12. extract_semantic_page

Extract structured semantic layout from HTML using DomDistiller-inspired algorithms. Produces a structured Control Map with headings, navigation, content blocks, forms, tables, and other semantic elements.

Returns:

{
  "content": [
    {
      "type": "text",
      "text": "{ \"success\": true, \"control_map\": { \"title\": \"...\", \"headings\": [...], \"navigation\": {...}, \"main_content\": [...], \"lists\": [...], \"forms\": [...], \"tables\": [...], \"media\": [...], \"footer\": {...} } }"
    }
  ]
}

13. generate_repo_graph

Generate repository structural map using Graphviz DOT and JSON formats. Analyzes file tree, classifies files by extension, builds dependency-like parent-child graph for deep codebase understanding.

Returns:

{
  "content": [
    {
      "type": "text",
      "text": "{ \"success\": true, \"graph\": { \"node_count\": 42, \"edge_count\": 38, \"nodes\": [...], \"edges\": [...] } }"
    },
    {
      "type": "text",
      "text": "--- Graphviz DOT ---\ndigraph repo {\n  rankdir=TB;\n  ...\n}"
    }
  ]
}

14. download_image

Download an image from a URL into the server's DOWNLOAD_DIR and return a download:// reference that can be passed to any other tool. Validates the URL, checks content-type, verifies image magic bytes, and enforces size limits.

Returns:

{
  "content": [
    {
      "type": "text",
      "text": "{ \"success\": true, \"downloadRef\": \"download://<uuid>.png\", \"file_path\": \"/abs/path/to/tmp/hvm-downloads/<uuid>.png\", \"original_url\": \"https://...\", \"mime_type\": \"image/png\", \"size\": 123456, \"width\": 800, \"height\": 600 }"
    }
  ]
}

Error scenarios:

• Invalid URL format (not http/https)

• DNS/network failure

• HTTP error (4xx/5xx)

• Wrong content-type (e.g., text/html)

• Payload does not contain valid image magic bytes

• Empty response

• File exceeds MAX_DOWNLOAD_SIZE_MB

---

## Image Input Formats

All tools accept image input through a resolution pipeline:

1. **Base64 Data URI**: `data:image/png;base64,<BASE64_STRING>`
2. **`file://` URI**: Converted to a local path on the server host.
3. **Home Directory Expansion**: `~/path` is expanded to the server's home directory.
4. **HTTP(S) URL**: Fetched, validated, and saved to `DOWNLOAD_DIR` before processing.
5. **Local Filesystem Path**: Resolved only if the file exists on the **server host** (not the client).
6. **Pure Base64 Fallback**: If the string length exceeds 50 characters, it is decoded and validated as an image.
7. **Upload Reference**: `upload://<filename>` — resolved from the server's `UPLOAD_DIR`.
8. **Download Reference**: `download://<filename>` — resolved from the server's `DOWNLOAD_DIR` (returned by the `download_image` tool).

> **Remote Host Rule**: The MCP Vision Server runs on a remote host. Local filesystem paths (e.g. `/tmp/...`, `C:\...`, `~/...`) are not visible to the server unless they exist on the server host. Transmit images as **Base64 Data URIs**, **HTTP URLs**, or **upload:// references**.

Supported formats: PNG, JPEG, GIF, WEBP, BMP.

All images are normalized to PNG via Sharp before OCR or AI analysis.

## Binary Response Streaming

To reduce MCP response payloads, tools that generate images save their output to `FEEDBACK_DIR` and include a file path in the JSON response text. The Base64 image is still included in the `image` content block for backward compatibility, but clients can switch to reading the file directly for large outputs.

| Tool | Output File Field | Description |
|------|------------------|-------------|
| `preprocess_and_crop` | `output_file_path` | Preprocessed PNG |
| `browser_screenshot_annotation` | `output_file_path` | Annotated PNG |
| `detect_ui_elements` | `overlay_file_path` | Overlay PNG (when `return_overlay=true`) |
| `visual_diff` | `diff_file_path` | Diff PNG |
| `textual_visual_feedback` | `screenshot.file_path` | Screenshot PNG |

## Testing

The project includes test scripts to validate end-to-end tool execution and edge-case handling.

### Quick Transport Test (`test-mcp.js`)

```bash
node test-mcp.js

Tests both Streamable HTTP (/mcp) and legacy SSE (/sse) transports with basic happy-path tool calls.

Comprehensive Edge-Case Test Runner (test_runner.mjs)

node test_runner.mjs

Generates synthetic images in memory using Sharp and runs a full matrix of happy-path and edge-case tests:

• check_vision_health: Ensures Ollama, Tesseract, and Sharp report status cleanly.

• fast_ocr_tesseract: Happy path with valid Base64; edge case with invalid/truncated Base64.

• preprocess_and_crop: Happy path with valid crop + grayscale; edge case with out-of-bounds crop.

• analyze_image: Happy path with prompt; edge case with empty prompt (default fallback).

• find_text_element: Happy path with valid query; edge case with missing query.

• compare_images: Happy path with 2+ images; edge case with single image (minimum length check).

• browser_screenshot_analysis: Happy path with focus/detail variants; happy path with default parameters.

• browser_screenshot_annotation: Happy path with label + box; edge case with empty annotations array and invalid annotation item; arrow + circle with return_base64=false.

• visual_diff: Happy path with two images and custom threshold/color; edge case with single image (minimum length check).

• detect_ui_elements: Happy path with no overlay; edge case with missing image_source.

• textual_visual_feedback: Happy path with OCR; happy path with DOM + CSS; edge case with missing image_source.

• extract_semantic_page: Happy path with comprehensive HTML; edge case with empty HTML.

• generate_repo_graph: Happy path on current repo; edge case with invalid path.

Upload Endpoint Test

# Upload a test image
curl -X POST http://localhost:11402/upload \
  --data-binary @test-image.png \
  -H "Content-Type: image/png"

# Expected response
# {"uploadRef":"upload://<uuid>.png","filename":"<uuid>.png","mimeType":"image/png","size":5242880,"width":1920,"height":1080}

Error Handling

All tool errors are caught and returned as structured MCP error responses. The server returns structured JSON inside content[0].text for machine-readable errors:

{
  "content": [
    {
      "type": "text",
      "text": "{ \"error\": true, \"code\": \"INVALID_ANNOTATION\", \"message\": \"Each annotation must be an object with a 'type' field.\", \"validTypes\": [\"label\",\"box\",\"arrow\",\"circle\"], \"suggestions\": [\"Ensure 'label' annotations include 'text'\", \"Ensure coordinate fields (x, y) are numbers\"] }"
    }
  ],
  "isError": true
}

Agent clients can parse the JSON to auto-recover or display actionable suggestions. Simple tool execution errors fall back to the original text response with isError: true.

Process-level guards (uncaughtException, unhandledRejection) are installed at server startup to prevent the Node.js process from crashing unexpectedly.

Utility Scripts

License

ISC