# screen-mcp

Screen-aware MCP (Model Context Protocol) server. Captures the user's screen, finds text-worthy regions via heuristic analysis, runs OCR, and returns a compact JSON (~500 tokens) instead of a full screenshot (~500,000 tokens).

## Architecture

| Module | Description |
|--------|-------------|
| screenshot.py | Fast screen capture via mss |
| detection.py | Heuristic text-region finder (edge density + contrast scoring) |
| ocr.py | Tesseract OCR wrapper |
| main.py | MCP stdio server + HTTP debug server + ScreenWatcher background monitor |

## Three Modes

### 1. MCP stdio (for Codex / Claude Desktop / Cursor)

Add to your MCP config (mcp.json):

`json
{
  "mcpServers": {
    "screen": {
      "command": "python",
      "args": ["-m", "src.main"],
      "cwd": "/path/to/screen-mcp"
    }
  }
}
`

Three tools are exposed:

- screen_analyze - Full analysis: capture -> detect regions -> OCR
- screen_watch - Background monitoring: start / stop / status / force_check
- screen_diff - One-shot change detection

### 2. Demo Mode

`ash
python -m src.main --demo
`

Outputs a JSON analysis of the current screen to stdout.

### 3. HTTP Debug Server

`ash
python -m src.main --http-port 8000
`

- GET /health returns {"status": "ok"}
- POST /analyze returns full screen analysis JSON

## Installation

`ash
pip install mss Pillow pytesseract numpy fastapi uvicorn

# Install Tesseract OCR separately
# Windows: winget install UB-Mannheim.TesseractOCR
# macOS: brew install tesseract
# Linux: sudo apt install tesseract-ocr
`

## Output Example

`json
{
  "monitors": [{"left": 0, "top": 0, "width": 1920, "height": 1080}],
  "regions": [{"x1": 420, "y1": 140, "x2": 1160, "y2": 460, "confidence": 0.76}],
  "texts": [
    {"text": "New chat", "confidence": 0.96, "bbox": [43, 54, 100, 64]},
    {"text": "Search", "confidence": 0.96, "bbox": [43, 84, 82, 94]}
  ],
  "summary": "Found 5 text regions | Text (118): New chat, Search, Plugins..."
}
`

## Token Savings

| Method | Tokens |
|--------|--------|
| Raw screenshot (1920x1080 PNG) | ~500,000+ |
| screen-mcp JSON | ~500-2,000 |
| Savings | 99.6% |

## Configuration

| Parameter | Default | Description |
|-----------|---------|-------------|
| --min-area | 3000 | Min pixel area for text region detection |
| --ocr-conf | 0.4 | Minimum OCR confidence threshold |
| --monitor | 1 | Monitor index (1=primary) |

## Notes

- OCR uses Tesseract with English language pack by default
- Chinese OCR requires installing chi_sim.traineddata
- The ScreenWatcher background monitor uses MSE-based frame differencing with configurable interval and threshold
- Tesseract auto-detects common installation paths; falls back to PATH lookup