image-recognition-mcp

An image recognition MCP server based on the macOS local Vision framework — letting vision-less AI models "see" screenshots and images.

Provides 4 MCP tools for AI clients (opencode / Claude Desktop / Cursor / Cline, etc.):
OCR text recognition / image subject classification / comprehensive recognition / screenshot and recognition. Fully local inference; data never leaves your machine.

Table of Contents

• Features

• Architecture

• Quick Start

• MCP Tools

• Input and Output Formats

• Trigger Mechanism

• Client Integration Configuration

• Performance and Resources

• Permissions and Privacy

• Troubleshooting

• Extension Suggestions

Related MCP server: npu-vision-fallback

Features

• 100% local inference: Based on the Apple Vision framework (VNRecognizeTextRequest + VNClassifyImageRequest), zero network requests, zero external API calls.

• Mixed Chinese-English OCR: Supports Chinese (zh-Hans), English, and 20+ languages, including handwriting recognition, with selectable accuracy levels (accurate / fast).

• Image subject/scene classification: Returns category labels with confidence scores, which the model can use to generate natural language descriptions.

• Three image sources: local path, data:image/png;base64,... URI, and raw base64 (PNG magic number validation).

• Automatic downscaling for large images: Images larger than 4096px are automatically downscaled before recognition, for faster speed and lower memory usage.

• Structured JSON output: All tools return a unified {status, ...} JSON, including confidence scores and normalized bounding boxes, making it easy for models to parse and reference.

• Optional screenshot: Directly invokes the screencapture command to capture and recognize the screen (requires screen recording permission).

Architecture

┌────────────────────────────────────────────────────────────┐
│  AI 会话客户端（opencode / Claude Desktop / Cursor / ...）   │
│  无视觉模型看到图片路径 → 调用工具                            │
└──────────────────────────┬─────────────────────────────────┘
                           │  MCP 协议 (stdio JSON-RPC)
┌──────────────────────────▼─────────────────────────────────┐
│  image-recognition MCP 服务器 (Python + MCPServer)          │
│  ┌──────────────┬──────────────┬──────────────┐            │
│  │  ocr_image   │recognize_image│describe_image│            │
│  │screenshot_…  │              │              │            │
│  └──────────────┴──────────────┴──────────────┘            │
└──────────────────────────┬─────────────────────────────────┘
                           │  Vision 框架调用 (pyobjc)
┌──────────────────────────▼─────────────────────────────────┐
│  macOS 本地视觉引擎                                          │
│  VNRecognizeTextRequest   —— OCR（中英+多语言）              │
│  VNClassifyImageRequest   —— 图像主体/场景分类                │
│  全程本机推理，无网络请求，数据不出设备                        │
└────────────────────────────────────────────────────────────┘

Quick Start

Requirements

• macOS 13+ (14+ recommended; the Vision framework delivers the best Chinese recognition results on it)

• Python 3.10+ (tested on 3.13.12)

• Xcode Command Line Tools installed (xcode-select --install)

Installation

# 克隆/进入项目目录
cd /path/to/image-recognition-mcp

# 创建 venv 并安装依赖
python3 -m venv .venv
source .venv/bin/activate
pip install -U pip
pip install -r requirements.txt

Self-test

# 生成一张含中英文的测试图片
.venv/bin/python scripts/make_test_image.py

# 直接测试 Vision 引擎（不走 MCP）
.venv/bin/python scripts/test_engine.py sample/test_card.png

# 端到端测试 MCP 服务器（启动 stdio，列出工具，调用 OCR）
.venv/bin/python scripts/test_mcp.py sample/test_card.png

Expected output: 3 lines of text (MacBook Air 图片识别测试 / Hello Vision OCR 12345 / 日期：2026-08-04 13:30) are fully recognized, and the image classification result is reasonable (document/printed_page/screenshot, etc.).

Direct command-line invocation of the engine (optional)

# OCR
.venv/bin/python vision_engine.py /path/to/image.png --mode ocr

# 主体分类
.venv/bin/python vision_engine.py /path/to/image.png --mode classify

# 综合识别
.venv/bin/python vision_engine.py /path/to/image.png --mode analyze

# 截屏到 ~/Pictures
.venv/bin/python vision_engine.py --mode shot

MCP Tools

After startup, the server exposes 4 tools to clients:

1. ocr_image — Extract text from an image (OCR)

{
  "image": "/Users/me/Pictures/shot.png",      // 必填，路径 / data URI / 纯 base64
  "languages": "zh-Hans,en-US",                // 可选，逗号分隔，顺序即优先级
  "min_confidence": 0.2,                       // 可选，0~1，过滤低置信度结果
  "filter_noise": true                         // 可选，默认 true，过滤图标/符号误识噪声
}

About filter_noise: Automatically filters out icon misrecognition noise in screenshots (such as •••, ③, a lone 8/凸, etc.), but keeps number strings that may carry business meaning (amounts, card numbers, transaction IDs, times, etc.). Filtered lines are placed separately in the returned noise field, so no information is lost; to get the raw full results, set filter_noise: false.

Returns:

{
  "status": "ok",
  "image": "/Users/me/Pictures/shot.png",
  "text": "完整拼接的全文",
  "count": 3,
  "lines": [
    {
      "text": "MacBook Air 图片识别测试",
      "confidence": 0.5,
      "bbox": {"x": 0.052, "y": 0.695, "width": 0.555, "height": 0.133}
    }
  ]
}

2. recognize_image — Comprehensive recognition

{
  "image": "/path/to/img.png",
  "languages": "zh-Hans,en-US"
}

Returns:

{
  "status": "ok",
  "image": "/path/to/img.png",
  "info": {"path": "...", "size_bytes": 12345, "pixel_width": 1200, "pixel_height": 420, "uti": "public.png"},
  "ocr": [...],
  "classification": [{"label": "document", "confidence": 0.529}, ...],
  "summary": "图中文字(OCR):\n... \n图像主体/场景: document(0.53)",
  "elapsed_ms": 98
}

3. describe_image — Subject/scene classification

{
  "image": "/path/to/img.png",
  "top_k": 8,                    // 1~20
  "min_confidence": 0.05
}

Returns:

{
  "status": "ok",
  "image": "/path/to/img.png",
  "labels": [
    {"label": "Animal", "confidence": 0.812},
    {"label": "Cat", "confidence": 0.703}
  ]
}

The label is in English (e.g., Animal / Landscape / Food / Vehicle); the calling model interprets and translates it as needed.

4. screenshot_and_recognize — Capture the screen and recognize

{
  "languages": "zh-Hans,en-US"
}

Captures the entire screen → OCR. Requires screen recording permission; see Permissions and Privacy.

Input and Output Formats

Input format (image parameter)

Measured in practice: A 256KB desktop screenshot → base64 data URI (~340K characters) → MCP tool call, recognizing 42 lines of valid text + 4 lines of noise in about 0.6s, with results identical to passing the path directly.

The server automatically:

• Validates that the path exists

• Decodes data URI / base64 and writes to a temporary file

• Validates format support (based on CGImageSource; compatible with JPEG/PNG/HEIC/TIFF/GIF/BMP/WebP)

Output format

• All tools return a string (JSON) so models can parse it directly.

• Success: {"status": "ok", ...}

• Failure: {"status": "error", "error": "..."}

• Bounding box coordinates (bbox) are normalized values (origin at bottom-left, 0~1), consistent with the Vision framework.

Trigger Mechanism

MCP uses a protocol design where "tools are invoked on demand by the model"; the server cannot proactively detect that the user has uploaded an image. To achieve "automatic triggering", coordination on the client/model side is required:

Trigger paths

Recommended prompt convention (critical)

To ensure 100% triggering, add the following to AGENTS.md in the project root or to the model's system prompt:

## 图片处理约定

当用户消息中出现图片路径或 base64 data URI 时：
- 若当前模型具备视觉能力 → 直接读取图片（不要调用本 MCP 服务器）。
- 若当前模型不具备视觉能力 → **必须**调用 `image-recognition` MCP 服务器的 `ocr_image` 或
  `recognize_image` 工具，将识别结果作为图片内容继续处理。
- 工具调用结果已经包含识别出的文字与图像描述，无需再要求用户提供说明。

Once this convention is written into AGENTS.md, clients such as opencode / Claude Desktop will send the instruction to the model as part of the system prompt, achieving true "automatic triggering".

Client Integration Configuration

Replace the absolute paths in the following configurations with the project location on your machine, then write them into the corresponding client's configuration file.

opencode

Write to opencode.json (project-level) or ~/.config/opencode/opencode.json (user-level):

{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "image-recognition": {
      "type": "local",
      "command": [
        "/path/to/image-recognition-mcp/.venv/bin/python",
        "/path/to/image-recognition-mcp/mcp_server.py"
      ],
      "enabled": true
    }
  }
}

After restarting opencode, you will see the 4 image-recognition tools in the tool list.

Claude Desktop

Write to ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "image-recognition": {
      "command": "/path/to/image-recognition-mcp/.venv/bin/python",
      "args": ["/path/to/image-recognition-mcp/mcp_server.py"]
    }
  }
}

Cursor / Cline / generic stdio MCP clients

{
  "mcpServers": {
    "image-recognition": {
      "command": "/path/to/image-recognition-mcp/.venv/bin/python",
      "args": ["/path/to/image-recognition-mcp/mcp_server.py"]
    }
  }
}

WorkBuddy

Edit ~/.workbuddy/mcp.json, add image-recognition to mcpServers, and restart for it to take effect:

WorkBuddy

See the configs/ directory for reference configuration examples:

• configs/opencode.example.json

• configs/claude-desktop.example.json

• configs/generic-stdio.example.json

Performance and Resources

Optimization tips:

• A 4096px auto-downscale is already built into _load_cg_image, which is sufficient for the vast majority of screenshots.

• When recognizing many images in batch, you can merge multiple ocr_image calls into a single recognize_image call on the client side to reduce context token consumption.

• Choosing level="fast" for OCR can speed things up by 30–50%, at the cost of slightly lower accuracy (small text, handwriting).

Permissions and Privacy

• Fully local: All recognition is done within the macOS Vision framework; data never leaves your machine, and no API Key or network is required.

• Screen recording permission (only required by the screenshot_and_recognize tool):

  • On first call, macOS will show a prompt or ask for authorization in "System Settings > Privacy & Security > Screen Recording".

  • Grant permission to the host process running the MCP server (e.g., Terminal, Claude Desktop, opencode).

  • Without authorization, the tool returns a clear error message rather than failing silently.

Troubleshooting

Extension Suggestions

To add more Vision capabilities, refer to the existing functions in vision_engine.py and add the corresponding Vision requests, for example:

• VNDetectFaceRectanglesRequest — Face detection

• VNGenerateAttentionBasedSaliencyImageRequest — Saliency regions

• VNDetectDocumentSegmentationRequest — Document region segmentation (scanning-type apps)

• VNRecognizeAnimalsRequest — Animal breed recognition (iOS 15+, macOS 12+)

After implementing, simply add a new @mcp.tool() in mcp_server.py to expose it to the model.

File Structure

image-recognition-mcp/
├── README.md                       # 本文档
├── requirements.txt                # Python 依赖
├── vision_engine.py                # Vision 框架封装（OCR + 分类 + 截图）
├── mcp_server.py                   # MCP 服务器主程序
├── scripts/
│   ├── make_test_image.py          # 生成含中英文的测试图片
│   ├── test_engine.py              # Vision 引擎自测
│   └── test_mcp.py                 # MCP 服务器端到端冒烟测试
├── configs/                        # 客户端配置示例
│   ├── opencode.example.json
│   ├── claude-desktop.example.json
│   └── generic-stdio.example.json
├── sample/
│   └── test_card.png               # 测试图片（含中文/英文/数字/红色圆形）
└── .venv/                          # Python 虚拟环境（运行后生成）

License

This project's code is under the MIT license. Vision framework calls are subject to the Apple SDK license and can only run on macOS.