🔋 npu-vision-fallback

Local low-power vision for desktop AI agents

When accessibility APIs fail — NPU-first, zero GPU wake-up, 100% local

English | 中文

What is this?

A lightweight, local-first vision service for desktop agents that need to see and interact with screens where traditional accessibility APIs fall short—games, remote desktops, canvas apps, and more.

Built for efficiency: Native OS OCR · Intel NPU acceleration · Zero cloud calls · Battery-friendly by design

✨ Why Use This?

Desktop agents face a challenge: how to perceive UI when the accessibility tree is empty?

Not a competitor to your cloud model — the gatekeeper in front of it. Modern multimodal LLMs are excellent at understanding a screen; they're wasteful when all the agent needs is "where is the Start button." npu-vision-fallback handles that low-level "find/read the widget" step locally — cheap, fast, offline — and you escalate to the big model only for the genuinely hard reasoning ("what does this dashboard mean?"). When the accessibility tree comes back empty, this is the layer that answers first, without touching the cloud or spinning up any GPU.

Perfect for:

• 🎮 Game UIs and emulators

• 🖥️ Remote desktop / VNC clients (no remote accessibility tree)

• 🎨 Canvas / WASM web apps rendering outside the DOM

• 💻 Local SLMs that can't afford multimodal screenshot tokens

💡 Design Philosophy

Three deliberate constraints — not limitations, but the whole point:

1. Don't use a sledgehammer to crack a nut. Try the OS-native path first (Windows OCR), then a small local detector. The cloud multimodal model is the last resort, reserved for real reasoning — not for reading a button label.

2. Isolate the compute. Let the NPU do inference so the GPU stays free to render the game/app the agent is looking at. By design this service requests NPU or CPU only — it never asks for any GPU, integrated or discrete (why?).

3. Local, period. Screenshots are captured in memory and never written to disk; OCR text is never logged; nothing leaves the machine.

The point isn't that cloud vision can't do this — it's that spending tokens, latency, and battery on "find the OK button" is the wrong tool for the job.

🚀 Quick Start

1. Install (Windows + Intel NPU recommended)

pip install "npu-vision-fallback[ocr-win,detect]"
python scripts/download_ui_model.py  # One-time setup

2. Configure your MCP client

Add to your MCP configuration (e.g., claude_desktop_config.json or equivalent):

{
  "mcpServers": {
    "npu-vision-fallback": {
      "command": "npu-vision-fallback"
    }
  }
}

3. Use it

Restart your agent and try:

You: The accessibility tree for this game is empty. Can you read the screen at coordinates [0,0,1280,800] and find the "Start Game" button?

Claude: (calls analyze_screen) I found a button labeled "Start Game" at [520, 580, 720, 640]. Want me to click its center at (620, 610)?

📦 Installation Options

Windows (Recommended)

Native OCR + NPU UI detection (~85 MB total):

pip install "npu-vision-fallback[ocr-win,detect]"
python scripts/download_ui_model.py

Linux / macOS

Cross-platform OCR + CPU detection (~130 MB):

pip install "npu-vision-fallback[ocr-rapid,detect]"
python scripts/download_ui_model.py

Full (All Backends)

For development or testing all backends:

pip install "npu-vision-fallback[all]"
python scripts/download_ui_model.py

Minimal Core

Just the MCP server (no OCR/detection, ~20 MB):

pip install npu-vision-fallback

💡 Note: The detect extra uses OpenVINO (~80 MB) for runtime, not PyTorch. Model conversion requires the dev-convert extra (~2 GB), but that's a one-time setup most users skip.

🎯 Key Features

• 🔋 NPU-first architecture — UI detection runs on Intel AI Boost at ~80ms per call (~0.3J energy)

• ⚡ Zero dGPU wake-up — Default paths use NPU, system OCR, or CPU—laptop battery stays happy

• 🌐 Native OS OCR — Uses Windows OCR engine (macOS Vision planned) for quality

• 🧩 MCP protocol — Works with Claude Code, Codex, OpenCode, Antigravity CLI, or any MCP client out of the box

• 🪶 Lightweight — No PyTorch/TensorFlow at runtime; all heavy deps are optional

• 🛡️ Privacy-first — 100% local processing, no telemetry, no cloud

⚡ Performance

Measured on Intel Core Ultra 9 275HX (2560×1600 screen, on battery):

Full benchmark details and reproduction steps: outputs/power_report.md

🛠️ MCP Tools

analyze_screen is the primary tool — it fuses detection + OCR, returns spatially-sorted elements with text annotations. Perfect for agent navigation.

analyze_image_file / analyze_image_directory reuse the exact same OCR/UI-detection/fusion logic as the screen tools, but operate on image files instead of live screen captures — useful for batch-processing screenshots or UI mockups you already have on disk.

💻 CLI

Besides the MCP server, a standalone CLI is available for local file/directory analysis:

# Single image
uv run npu-vision-fallback-cli analyze-image path/to/image.png --mode all

# Directory batch (writes result JSON per image + manifest.json to --output-dir)
uv run npu-vision-fallback-cli analyze-dir path/to/dir --output-dir out/ --recursive

📚 Documentation

• Architecture Guide — System design and data flow

• Backend Reference — Per-backend capabilities and priorities

• FAQ — Common questions and troubleshooting

• Contributing — How to contribute

• Code Guide — Project constitution for contributors

🧪 Examples

uv run python examples/basic_ocr.py --region 0 0 1280 800

🗺️ Roadmap

• v1.1 — Multi-monitor support, DPI scaling awareness

• v2.0 — Custom model training interface, bring your own detector

• v2.1 — UI-TARS integration, macOS Vision backend, PP-OCR v4 on NPU

🤝 Contributing

Contributions welcome! See CONTRIBUTING.md for guidelines. Please read CLAUDE.md—it's the project constitution that ensures code quality and architectural consistency.

📋 Supported Backends

📄 License

MIT © npu-vision-fallback contributors

🙏 Acknowledgments

Built with:

• Model Context Protocol (Anthropic) — Agent integration layer

• OpenVINO — NPU/CPU inference runtime

• Ultralytics YOLO — UI detection models

• RapidOCR — Cross-platform OCR engine

• Tesseract — OCR fallback

• python-mss — Screen capture library

Development assisted by Claude Code (Anthropic). Architecture design and code review powered by AI collaboration.