annotation-mcp

MCP (Model Context Protocol) server for image annotation. Draws bounding boxes, arrows, highlights, callouts, text, and circles on images. Also detects barcodes and text regions, and crops image regions for iterative inspection.

Supported Platforms

Related MCP server: EasyOCR MCP Server

Tools

Installation

1. System dependencies

Tesseract OCR and zbar barcode library are required for detect_text_regions and detect_barcodes. The other 5 tools (annotate, draw_bounding_box, etc.) work without them.

macOS

brew install tesseract tesseract-lang zbar

This installs:

• tesseract (OCR engine)

• tesseract-lang (163 language packs including eng, rus, jpn, chi_sim, deu, fra, etc.)

• zbar (barcode scanning library, including libzbar.dylib)

Verify:

tesseract --version
tesseract --list-langs   # should show 'eng' and 'rus'

Linux (Debian/Ubuntu)

sudo apt-get update
sudo apt-get install -y \
  tesseract-ocr \
  tesseract-ocr-eng \
  tesseract-ocr-rus \
  libzbar0 \
  zbar-tools

For Fedora / RHEL:

sudo dnf install -y tesseract tesseract-langpack-eng tesseract-langpack-rus zbar

For Arch:

sudo pacman -S tesseract tesseract-data-eng zbar

Verify:

tesseract --version
tesseract --list-langs

Windows

The recommended install is via Chocolatey:

choco install --no-progress tesseract -y
choco install --no-progress python3 -y  # if you don't have Python yet

After install, ensure C:\Program Files\Tesseract-OCR is on your PATH (the choco package should do this automatically; you may need to restart your terminal).

pyzbar ships its own copy of libzbar-0.dll bundled in the wheel on Windows, so no separate zbar install is needed.

Verify in PowerShell:

tesseract --version
tesseract --list-langs

2. Python package

The recommended Python install uses uv (fast, hermetic).

macOS / Linux

# Install via uv (creates an isolated tool install)
uv tool install annotation-mcp

# Or with pipx (similar to uv tool, classic alternative)
pipx install annotation-mcp

# Or with pip
pip install annotation-mcp

The annotation-mcp command will be installed and the server can be registered in your MCP client config (e.g. ~/.config/opencode/opencode.json):

{
  "mcp": {
    "annotation": {
      "type": "local",
      "command": ["annotation-mcp"],
      "enabled": true
    }
  }
}

Windows

# With uv (recommended)
uv tool install annotation-mcp

# With pip
pip install annotation-mcp

The annotation-mcp.exe will be installed. Same MCP client config works.

3. Install from source (development)

git clone https://github.com/aschokinatgmail/annotation-mcp.git
cd annotation-mcp
uv sync --all-extras --dev
uv tool install --force --reinstall .

Detection tools — workflow

The detection tools are designed to solve the "where is X in this image?" problem. They return structured text + bbox data so the AI can match semantic labels to coordinates deterministically, rather than guessing from approximate vision.

Typical workflow

1. Call detect_text_regions to get all text regions in the image with their content, bounding boxes, and confidence scores.

2. Match semantic label to region by inspecting the text field. E.g., for "find the brand title", look for the region whose text matches the brand name.

3. Refine if needed — if a region is split across multiple detections (e.g., the brand title is "ФИТОС" + "ПОРИН"), use the crop_for_inspection tool to zoom in and re-detect, or extend the bbox using a y-band heuristic.

4. Call annotate with the discovered bboxes.

Example: annotating a product package

# Step 1: Find all text regions
text_matches = detect_text_regions(image_path, lang="eng+rus", detail="word", min_confidence=30)
# Returns 400+ regions with text content + bbox_pixels

# Step 2: Find the brand title region
brand_region = next(r for r in text_matches if "ФИТО" in r["text"].upper())
# Returns: {text: "ФИТОС", bbox_pixels: [1019, 786, 1300, 881], confidence: 88, ...}

# Step 3: Find the barcode (deterministic decoding)
barcodes = detect_barcodes(image_path)
# Returns: [{type: "EAN13", data: "4607026420155", bbox_pixels: [2366, 3403, 2622, 3994], ...}]

# Step 4: Find the dosage table by keyword
table_regions = [r for r in text_matches if any(kw in r["text"].lower() for kw in ["мл", "м²", "капля", "растен"])]
# Take the union bbox

# Step 5: Annotate
annotate(image_path, output_path="result.png", normalized=True, annotations=[
    {"type": "bbox", "bbox": brand_region["bbox_normalized"], "label": "Brand", "color": "#FF0000"},
    {"type": "bbox", "bbox": barcodes[0]["bbox_normalized"], "label": "Barcode", "color": "#00AA00"},
    {"type": "bbox", "bbox": table_bbox_normalized, "label": "Table", "color": "#0066FF"},
])

When tesseract quality is poor: re-recognize with a vision model

Tesseract produces reliable output on clean documents but can return high-confidence nonsense on real-world photos (small, low-contrast, curved, embossed text). The detect_text_regions tool has two features for handling this:

• filter_garbage (default True): rejects OCR-noise regions (e.g. VLOAOL, yOLO, ||||) based on a heuristic that combines text length, character-class diversity, vowel ratio, dictionary presence, and part-number pattern. Catches ~98% of typical Tesseract garbage on real photos.

• crop_regions (default False): when set to True, the tool also writes per-region image crops to disk and returns their paths in the result. A vision model can then re-recognize each crop with higher accuracy. The workflow:

# Step 1: get tesseract bboxes + cropped images for each region
result = handle_detect_text_regions(
    photo_path,
    lang="eng", detail="word", preprocess="clahe",
    crop_regions=True,
    crop_padding=0.15,
)
regions = result.structuredContent["regions"]
crops = result.structuredContent["region_crops"]

# Step 2: for each crop, send to vision model for re-recognition
for crop in crops:
    vision_text = recognize_with_vision_model(crop["crop_path"])
    # vision_text is the actual text in that region

When a region is split across multiple detections

OCR can split a single visual line into multiple text regions (especially with stylized fonts or low-resolution scans). To recover the full region:

1. Find a single word region whose text you recognize.

2. Compute the y-band: y_band = (bbox_bottom - bbox_top) * 0.6 around the y-center.

3. Find all regions in the same y-band.

4. Take the union bbox (min/max of x1,y1,x2,y2).

5. Optionally add padding.

Preprocessing modes for detect_text_regions

If opencv-python is not installed, only none is available; the others fall back to PIL autocontrast.

Graceful degradation

If system dependencies are missing, the detection tools return clear error messages instead of crashing:

• detect_barcodes without pyzbar/zbar: "pyzbar not installed. Install with: pip install pyzbar; and the zbar shared library: brew install zbar (macOS) or apt install libzbar0 (Linux)"

• detect_text_regions without tesseract: "tesseract binary not found. Install with: brew install tesseract tesseract-lang"

• detect_text_regions with missing language pack: "Tesseract language pack(s) not installed: ['rus']. Install with: brew install tesseract-lang"

The 5 original annotation tools (annotate, draw_bounding_box, highlight_region, draw_numbered_callouts, get_image_info) work without any system dependencies.

HEIC / iPhone photo support

iPhone photos are stored as .HEIC files. The image_io module registers pillow_heif at import time so PIL can decode HEIC directly. This works on all three platforms with no extra setup.

HEIC test fixtures are not committed to the repo (HEIC files can contain EXIF metadata with GPS coordinates and device info). The HEIC tests skip cleanly when no local HEIC file is found. To exercise them locally, place a HEIC file at any of:

• tests/fixtures/exif_orientation.heic

• tests/fixtures/sample.heic

• ~/Pictures/sample.heic

Coordinate systems

All tools accept either pixel coordinates ([x1, y1, x2, y2] in absolute pixels) or normalized coordinates ([0.0, 0.0, 1.0, 1.0] fractions of image dimensions). Set the normalized flag accordingly.

Output bboxes are always provided in both pixel and normalized form for the caller's convenience.

Development

# Clone and install dev dependencies
git clone https://github.com/aschokinatgmail/annotation-mcp.git
cd annotation-mcp
uv sync --all-extras --dev

# Run tests (cross-platform)
uv run pytest

# Run tests with coverage
uv run pytest --cov=annotation_mcp

# Type check (strict mode)
uv run basedpyright

# Lint
uv run ruff check

CI runs on every push and PR across macOS, Linux, and Windows with Python 3.12: .github/workflows/ci.yml

Architecture

• image_io.py — Image loading with EXIF orientation handling (all 8 spec values), HEIC support.

• process.py / render.py — Annotation validation and PNG rendering.

• coords.py — Coordinate conversion (pixel ↔ normalized).

• handlers.py / handlers_detection.py — MCP tool handlers.

• server.py — FastMCP server registration (8 tools).

• detection/ — Barcode (pyzbar), OCR (tesseract + garbage filter), and crop modules.

Privacy & security notes

• EXIF data: Real photos often contain GPS coordinates, device info, and timestamps. The test fixtures in this repo have all such metadata stripped. The crop_for_inspection and annotate tools DO NOT strip EXIF from output images by default — if you save annotated images, the EXIF from the source is preserved. Strip it explicitly with PIL or exiftool if needed.

• No telemetry: This server does not collect or transmit any data. All processing is local.

• No file uploads: The server reads images from the local filesystem at the path you provide. It does not fetch images from URLs.

License

MIT.