🔋 npu-vision-fallback

Local low-power vision for desktop AI agents

When accessibility APIs fail — NPU-first, zero GPU wake-up, 100% local

English | 中文文档

English

What is this?

A lightweight, local-first vision service for desktop agents that need to see and interact with screens where traditional accessibility APIs fall short—games, remote desktops, canvas apps, and more.

Built for efficiency: Native OS OCR · Intel NPU acceleration · Zero cloud calls · Battery-friendly by design

✨ Why Use This?

Desktop agents face a challenge: how to perceive UI when the accessibility tree is empty?

npu-vision-fallback is your fallback layer — when the accessibility tree comes back empty, this gives your agent a small, fast, local vision service that doesn't touch the cloud or spin up the dGPU.

Perfect for:

• 🎮 Game UIs and emulators

• 🖥️ Remote desktop / VNC clients (no remote accessibility tree)

• 🎨 Canvas / WASM web apps rendering outside the DOM

• 💻 Local SLMs that can't afford multimodal screenshot tokens

🚀 Quick Start

1. Install (Windows + Intel NPU recommended)

pip install "npu-vision-fallback[ocr-win,detect]"
python scripts/download_ui_model.py  # One-time setup

2. Configure Claude Desktop

Add to your claude_desktop_config.json:

{
  "mcpServers": {
    "npu-vision-fallback": {
      "command": "npu-vision-fallback"
    }
  }
}

3. Use it

Restart Claude Desktop and try:

You: The accessibility tree for this game is empty. Can you read the screen at coordinates [0,0,1280,800] and find the "Start Game" button?

Claude: (calls analyze_screen) I found a button labeled "Start Game" at [520, 580, 720, 640]. Want me to click its center at (620, 610)?

📦 Installation Options

Windows (Recommended)

Native OCR + NPU UI detection (~85 MB total):

pip install "npu-vision-fallback[ocr-win,detect]"
python scripts/download_ui_model.py

Linux / macOS

Cross-platform OCR + CPU detection (~130 MB):

pip install "npu-vision-fallback[ocr-rapid,detect]"
python scripts/download_ui_model.py

Full (All Backends)

For development or testing all backends:

pip install "npu-vision-fallback[all]"
python scripts/download_ui_model.py

Minimal Core

Just the MCP server (no OCR/detection, ~20 MB):

pip install npu-vision-fallback

💡 Note: The detect extra uses OpenVINO (~80 MB) for runtime, not PyTorch. Model conversion requires the dev-convert extra (~2 GB), but that's a one-time setup most users skip.

🎯 Key Features

• 🔋 NPU-first architecture — UI detection runs on Intel AI Boost at ~80ms per call (~0.3J energy)

• ⚡ Zero dGPU wake-up — Default paths use NPU, system OCR, or CPU—laptop battery stays happy

• 🌐 Native OS OCR — Uses Windows OCR engine (macOS Vision planned) for quality

• 🧩 MCP protocol — Works with Claude Desktop, Cursor, or any MCP client out of the box

• 🪶 Lightweight — No PyTorch/TensorFlow at runtime; all heavy deps are optional

• 🛡️ Privacy-first — 100% local processing, no telemetry, no cloud

⚡ Performance

Measured on Intel Core Ultra 9 275HX (2560×1600 screen, on battery):

Full benchmark details and reproduction steps: outputs/power_report.md

🛠️ MCP Tools

analyze_screen is the primary tool — it fuses detection + OCR, returns spatially-sorted elements with text annotations. Perfect for agent navigation.

📚 Documentation

• Architecture Guide — System design and data flow

• Backend Reference — Per-backend capabilities and priorities

• FAQ — Common questions and troubleshooting

• Contributing — How to contribute

• Code Guide — Project constitution for contributors

🧪 Examples

uv run python examples/basic_ocr.py --region 0 0 1280 800

🗺️ Roadmap

• v1.1 — Multi-monitor support, DPI scaling awareness

• v2.0 — Custom model training interface, bring your own detector

• v2.1 — UI-TARS integration, macOS Vision backend, PP-OCR v4 on NPU

🤝 Contributing

Contributions welcome! See CONTRIBUTING.md for guidelines. Please read CLAUDE.md—it's the project constitution that ensures code quality and architectural consistency.

📋 Supported Backends

📄 License

MIT © npu-vision-fallback contributors

🙏 Acknowledgments

Built with:

• Model Context Protocol (Anthropic) — Agent integration layer

• OpenVINO — NPU/CPU inference runtime

• Ultralytics YOLO — UI detection models

• RapidOCR — Cross-platform OCR engine

• Tesseract — OCR fallback

• python-mss — Screen capture library

Development assisted by Claude Code (Anthropic). Architecture design and code review powered by AI collaboration.