# Unlimited-OCR MCP Server

MCP (Model Context Protocol) Server and Tool definitions for [Unlimited-OCR](https://github.com/baidu/Unlimited-OCR) — enables Agent frameworks (LangChain, DuMate, AutoGPT, CrewAI, etc.) to invoke OCR as a structured tool.

## Features

- **MCP Server** — exposes OCR capabilities via the Model Context Protocol (stdio & SSE transport)
- **OpenAI-compatible Tool Definitions** — JSON schemas for function calling
- **Dual Backend** — supports both local Transformers inference and remote vLLM/SGLang HTTP API
- **Three OCR Modes** — single image (`gundam` / `base`), multi-image, and PDF
- **Full Parameter Coverage** — `crop_mode`, `ngram`, `max_length`, `dpi`, etc.

## Quick Start

### Install

```bash
pip install -e ".[all]"
# Or minimal install (HTTP backend only):
pip install -e ".[pdf]"
```

### Start MCP Server

#### Option 1: HTTP backend (vLLM / SGLang server)

```bash
# Start an SGLang/vLLM server first
python -m sglang.launch_server \
    --model baidu/Unlimited-OCR \
    --served-model-name Unlimited-OCR \
    --port 10000 ...

# Then start the MCP server
UNLIMITED_OCR_BACKEND=http \
UNLIMITED_OCR_SERVER_URL=http://127.0.0.1:10000 \
python -m mcp.server
```

#### Option 2: Transformers backend (local GPU)

```bash
UNLIMITED_OCR_BACKEND=transformers \
UNLIMITED_OCR_MODEL=baidu/Unlimited-OCR \
python -m mcp.server
```

#### Option 3: SSE transport (for remote agents)

```bash
UNLIMITED_OCR_BACKEND=http \
python -m mcp.server --transport sse --port 8080
```

### Configure in Agent Frameworks

#### Claude Desktop / MCP Client

Add to your MCP client config:

```json
{
  "mcpServers": {
    "unlimited-ocr": {
      "command": "python",
      "args": ["-m", "mcp.server"],
      "env": {
        "UNLIMITED_OCR_BACKEND": "http",
        "UNLIMITED_OCR_SERVER_URL": "http://127.0.0.1:10000"
      }
    }
  }
}
```

#### LangChain (direct tool call)

```python
from skill.tool_definitions import call_tool

# Direct call — no LLM orchestrator needed
result = call_tool("ocr_parse_image", {
    "image_path": "document.jpg",
    "image_mode": "gundam",
})
print(result["text"])
```

#### LangChain (LLM tool calling)

```python
from skill.tool_definitions import get_openai_tools
from mcp.ocr_engine import OCREngine

tools = get_openai_tools()
llm_with_tools = llm.bind_tools(tools)

# The LLM decides when to call OCR
response = llm_with_tools.invoke("Parse the document at document.jpg")
```

See `examples/langchain_integration.py` for a complete example.

## Available Tools

### `ocr_parse_image`

Parse a single image using Unlimited-OCR.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `image_path` | string | *required* | Path to the image file |
| `image_mode` | `gundam` \| `base` | `gundam` | Image processing mode |
| `output_dir` | string | — | Directory to save results |
| `no_repeat_ngram_size` | int | 35 | N-gram repetition penalty |
| `ngram_window` | int | 128 | N-gram window size |
| `max_length` | int | 32768 | Max output token length |

**Image modes:**
- `gundam` (default): `base_size=1024`, `image_size=640`, `crop_mode=True` — recommended for most images
- `base`: `base_size=1024`, `image_size=1024`, `crop_mode=False` — full image processing

### `ocr_parse_multi`

Parse multiple images (multi-page). Always uses `base` mode.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `image_paths` | string[] | *required* | List of image paths |
| `output_dir` | string | — | Directory to save results |
| `no_repeat_ngram_size` | int | 35 | N-gram repetition penalty |
| `ngram_window` | int | 1024 | N-gram window size |
| `max_length` | int | 32768 | Max output token length |

### `ocr_parse_pdf`

Parse a PDF document (auto-converts pages to images, then multi-page OCR).

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `pdf_path` | string | *required* | Path to the PDF file |
| `dpi` | int | 300 | DPI for PDF-to-image conversion |
| `output_dir` | string | — | Directory to save results |
| `no_repeat_ngram_size` | int | 35 | N-gram repetition penalty |
| `ngram_window` | int | 1024 | N-gram window size |
| `max_length` | int | 32768 | Max output token length |

## Configuration

Environment variables:

| Variable | Default | Description |
|----------|---------|-------------|
| `UNLIMITED_OCR_BACKEND` | `transformers` | Backend: `transformers` or `http` |
| `UNLIMITED_OCR_MODEL` | `baidu/Unlimited-OCR` | Model name or path |
| `UNLIMITED_OCR_SERVER_URL` | `http://127.0.0.1:10000` | vLLM/SGLang server URL |
| `UNLIMITED_OCR_TORCH_DTYPE` | `bfloat16` | Torch dtype (transformers only) |
| `UNLIMITED_OCR_DEVICE` | `cuda` | Device (transformers only) |

## Project Structure

```
unlimited-ocr-mcp/
├── mcp/                         # MCP Server
│   ├── __init__.py
│   ├── ocr_engine.py           # OCR engine (Transformers + HTTP backends)
│   ├── server.py                # MCP Server (stdio + SSE transport)
│   └── pdf_utils.py             # PDF to image conversion
├── skill/                       # Skill / Tool definitions
│   ├── _meta.json               # Skill metadata
│   ├── SKILL.md                 # Skill definition (name, description, instructions)
│   ├── skill-card.md            # Skill card (publisher, use case, risks)
│   ├── tool_definitions.py      # OpenAI-compatible tool definitions (Python)
│   └── openai_tools.json        # Standalone tool schema (JSON)
├── examples/
│   ├── langchain_integration.py
│   └── mcp_client_example.py
├── pyproject.toml
├── requirements.txt
├── LICENSE
└── README.md
```

## Requirements

- Python >= 3.10
- **MCP backend**: `mcp` SDK >= 1.0.0
- **Transformers backend**: `torch`, `transformers`, `Pillow`, `einops`, `addict`, `easydict`
- **HTTP backend**: `requests` (vLLM or SGLang server running separately)
- **PDF support**: `pymupdf` >= 1.23.0

## Related

- [Unlimited-OCR](https://github.com/baidu/Unlimited-OCR) — the upstream OCR model
- [MCP Specification](https://modelcontextprotocol.io/) — Model Context Protocol docs
- [Feature Request Issue #97](https://github.com/baidu/Unlimited-OCR/issues/97) — the original proposal

## License

Apache License 2.0