🔋 npu-vision-fallback

⚠️ Archived experiment — no longer actively developed

An Intel NPU screen-vision + MCP integration experiment. Backend routing and Windows-native OCR integration worked; the NPU-first product hypothesis did not find a stable user need. Code preserved for reference.

English | 中文

🗄️ Why This Is Archived

This project tested whether desktop AI agents would benefit from running low-level screen perception — "find the button", "read this region" — locally on an Intel NPU, exposed as an MCP server. That was a hypothesis, not a verified need; it never crystallized into stable demand, so development stopped.

Two things the experiment did prove:

• Backend routing with graceful fallback — NPU → CPU → system OCR → cross-platform OCR, driven by a power policy, with heavy dependencies kept optional.

• Windows-native OCR over MCP — WinRT OCR wired into the same tool surface as the OpenVINO NPU UI detector.

What I still use it for: batch OCR of long screenshots. Note that the implementation has no complete tiling guarantee for very long images — it may drop content on extreme captures. Treat the code as a reference, not a maintained dependency.

🧱 Original Design

Three constraints that shaped the project:

1. Cheapest path first. OS-native OCR, then a local detector. A cloud multimodal model was the last resort — for reasoning, not for finding buttons.

2. Isolate compute. Inference on the NPU so the GPU stays free for the app the agent is watching. The service requests NPU or CPU only — never any GPU.

3. Local, period. Screenshots stay in memory and are never written to disk; OCR text is never logged; nothing leaves the machine.

Pipeline: mcp_server.py → core/orchestrator.py → core/backend_selector.py → selected backend → screen-space remap. Full detail: docs/ARCHITECTURE.md.

📋 Reference

MCP Tools

analyze_screen fuses detection + OCR into spatially-sorted elements with text annotations. analyze_image_file and analyze_image_directory reuse the same pipeline on image files.

CLI

# Single image
uv run npu-vision-fallback-cli analyze-image path/to/image.png --mode all

# Directory batch
uv run npu-vision-fallback-cli analyze-dir path/to/dir --output-dir out/ --recursive

Installation (reference only — not recommended for new deployments)

pip install "npu-vision-fallback[ocr-win,detect]"
python scripts/download_ui_model.py  # one-time model export to OpenVINO IR

Further extras are listed in pyproject.toml.

Supported Backends

Measured Performance

Intel Core Ultra 9 275HX, 2560×1600, on battery:

Full details: outputs/power_report.md

Examples

uv run python examples/basic_ocr.py --region 0 0 1280 800

Docs

• Architecture · Backends · FAQ

• Changelog · Code Guide

📄 License

MIT © npu-vision-fallback contributors

🙏 Acknowledgments

Built with MCP (Anthropic), OpenVINO, Ultralytics YOLO, RapidOCR, Tesseract, and python-mss.

Development assisted by Claude Code (Anthropic).