# OCR MCP

An **HTTP (Streamable) MCP server** written in Node.js that receives a file and a
structured JSON object with empty values, extracts the file's content using **local
OCR / text extraction** (no cloud, no API keys), and fills in the JSON values.

Supported file types:

| Type  | How it's read                                                        |
| ----- | -------------------------------------------------------------------- |
| Image | Tesseract OCR (`tesseract.js`) — png, jpg, gif, bmp, tiff, webp      |
| PDF   | Embedded text (`pdf-parse`); falls back to rendering + OCR if scanned |
| DOCX  | Direct text extraction (`mammoth`)                                   |
| DOC   | Not supported (convert to DOCX/PDF/image)                            |

## How field filling works

Filling happens in two passes:

1. **Heuristic parser (always on, local).** Matches each JSON key to a label in the
   document text, e.g. a key `invoiceNumber` matches lines like `Invoice Number: INV-2026`
   or `invoice_number - INV-2026`. Nested objects are supported.
2. **LLM fallback (optional).** For any field the heuristic could not resolve, a
   configured LLM is asked to fill the remaining values from the extracted text. It only
   fills still-empty fields and never overrides values already found. Enable it via `.env`
   (see below); leave it disabled to stay fully local.

Fields that remain empty after both passes are returned in an `unfilled` list.

## Requirements

- Node.js 18+ (developed on v24)

## Install

```bash
npm install
cp .env.example .env   # then edit .env if you want the LLM fallback
```

## Configuration (.env)

| Variable       | Description                                                        |
| -------------- | ------------------------------------------------------------------ |
| `PORT`         | Server port (default `3000`).                                      |
| `MCP_PATH`     | MCP endpoint path (default `/mcp`).                                |
| `LLM_PROVIDER` | `openai` \| `anthropic` \| `ollama`. Empty = LLM fallback disabled. |
| `LLM_MODEL`    | Model name, e.g. `gpt-4o-mini`, `claude-3-5-sonnet-latest`, `llama3.1`. |
| `LLM_API_KEY`  | Provider API key (not required for `ollama`).                      |
| `LLM_BASE_URL` | Optional base URL override.                                        |

Provider defaults for `LLM_BASE_URL`:

- `openai` → `https://api.openai.com/v1`
- `anthropic` → `https://api.anthropic.com`
- `ollama` → `http://localhost:11434/v1`

`openai` and `ollama` use the OpenAI-compatible Chat Completions API; `anthropic` uses
the Messages API. Any OpenAI-compatible endpoint (Azure OpenAI, LM Studio, vLLM, etc.)
works by pointing `LLM_PROVIDER=openai` at a custom `LLM_BASE_URL`.

## Run

```bash
npm start
# OCR MCP server (Streamable HTTP) listening on http://localhost:3000/mcp
# LLM fallback enabled: openai / gpt-4o-mini   (or "disabled")
```

Health check: `GET /health`.

## MCP tool

### `fill_json_from_file`

| Argument     | Type   | Required | Description                                                        |
| ------------ | ------ | -------- | ------------------------------------------------------------------ |
| `fileBase64` | string | yes      | File content encoded as base64.                                    |
| `mimeType`   | string | one of\* | MIME type, e.g. `image/png`, `application/pdf`.                    |
| `filename`   | string | one of\* | Original filename, used to detect the type.                        |
| `schema`     | object | yes      | JSON object with the desired keys and **empty** values.            |
| `language`   | string | no       | Tesseract language code(s), e.g. `ind+eng` or `eng+fra` (default `ind+eng`).|

\* Provide at least one of `mimeType` or `filename`.

**Returns** JSON (as text + `structuredContent`):

```json
{
  "data": { "name": "Acme Corp", "invoiceNumber": "INV-2026", "missingField": "" },
  "unfilled": ["missingField"],
  "extractedTextLength": 41,
  "usedLlm": false
}
```

`usedLlm` indicates whether the LLM fallback was invoked. If a fallback call fails, an
`llmError` field is included and the heuristic result is still returned.

## Connecting from an MCP client

Point a Streamable-HTTP-capable MCP client at `http://localhost:3000/mcp`. Example VS
Code `mcp.json`:

```json
{
  "servers": {
    "ocr": {
      "type": "http",
      "url": "http://localhost:3000/mcp"
    }
  }
}
```

## Example (raw HTTP)

```bash
# 1. initialize -> read the mcp-session-id response header
# 2. POST notifications/initialized with that session id
# 3. POST tools/call with the base64 file + schema
# See the curl handshake used in development; Accept must include
# "application/json, text/event-stream".
```