Skip to main content
Glama

OmniSight

繁體中文說明:README.zh-TW.md

Computer use for every Windows app window, for any MCP agent: GPU-rendered, covered or in the background.

Agents that drive a desktop through screenshots keep hitting the same walls on Windows:

  • GPU-rendered windows (games, emulators, Chromium / Electron, UWP) come back black from classic GDI capture.

  • A window that is covered or in the background is not in a full-screen screenshot at all.

  • Every step sends a full-screen image to a vision model just to find one button, which is slow and expensive.

  • Clicks land in the wrong place on scaled and mixed-DPI monitors.

OmniSight is an MCP server that fixes these at the source. Claude Code, Codex, Cursor, Cline, Claude Desktop and any other MCP client can use it.

How it works

  1. Structure first, pixels second. By default a snapshot returns the UI Automation element tree: every element has an id, a position and the actions it supports. The whole tree comes back in one cross-process call (a UIA cache request), typically in 30–300 ms, and it is text, not an image.

  2. Act on elements without the mouse. click(element="e7") runs the UIA Invoke / Toggle / Select pattern and type_text(replace=true) sets the value directly. This works on background windows and does not steal focus.

  3. Capture one window, with backends that fall back automatically. printwindow → wgc (Windows.Graphics.Capture) → bitblt → screen. A black or single-colour frame counts as a failure and the next backend is tried. The backend that worked is remembered per process.

  4. Image and OCR only when the tree is not enough. In auto mode a screenshot and OCR text with positions are added when the tree does not describe the content (games, emulators, canvas apps, custom-drawn UIs).

  5. One coordinate space. Element positions, OCR boxes and the image all use window-relative pixels of the snapshot image, so a position read from any of them can go straight into click. The process is per-monitor DPI aware.

Measured on a Windows 10 machine: a covered emulator window captured in about 100 ms, a 153-element Discord tree in about 130 ms, and repeat captures of a covered Discord window in under 200 ms end to end over MCP.

Related MCP server: windows-computer-use-mcp

Tools

Tool

What it does

list_windows

Windows front to back: hwnd, process, title, size, and flags such as minimized, admin, capture-blocked

snapshot

mode = tree / image / ocr / auto / all

click

Click an element id or an x,y; background=true posts messages instead of moving the mouse

type_text

Type text in any language, or set a control's value directly

press_key

Key combinations such as ctrl+s; scancode=true for games that read hardware scan codes

scroll / drag

Scroll and drag

element_action

invoke / toggle / expand / select / set_value / focus

find_element

Find elements by text

wait_for_change

Wait until the window's pixels change (loading, animations)

screenshot_screen

Whole-screen capture, for cross-window context

focus_window

Bring a window to the front

The window argument takes an hwnd, or part of the title or process name.

Install

Requires Windows 10 1903 or later and uv. Nothing to install up front: uvx fetches OmniSight from PyPI the first time the agent starts it.

The [ocr] extra adds RapidOCR (about 70 MB). It is far more accurate than the built-in Windows OCR on small UI fonts and mixed Chinese / English text. Without it, use uvx omnisight and OmniSight falls back to Windows OCR.

Connect an agent

Claude Code

claude mcp add -s user omnisight -- uvx --from "omnisight[ocr]" omnisight

Codex (~/.codex/config.toml)

[mcp_servers.omnisight]
command = "uvx"
args = ["--from", "omnisight[ocr]", "omnisight"]

Cursor, Cline, Claude Desktop and other mcpServers JSON configs

{
  "mcpServers": {
    "omnisight": {
      "command": "uvx",
      "args": ["--from", "omnisight[ocr]", "omnisight"]
    }
  }
}

From source

git clone https://github.com/Alex20090906/omnisight
cd omnisight
uv sync --extra ocr
claude mcp add -s user omnisight -- uv run --directory <path> omnisight

Environment variables

Variable

Meaning

OMNISIGHT_OCR_ENGINE

auto (default: RapidOCR if installed), rapid or windows

OMNISIGHT_OCR_LANG

Language for Windows OCR, such as zh-Hant-TW or en-US

What it deliberately does not do

  • It does not bypass SetWindowDisplayAffinity. Password managers, banking apps and DRM video opt out of capture on purpose. OmniSight says so clearly and suggests the element tree, instead of returning a black image the agent would retry forever.

  • It does not evade anti-cheat or anti-bot detection, and it does not touch the UAC secure desktop or the lock screen.

  • Admin windows: Windows (UIPI) drops input sent to an elevated window from a non-elevated process. OmniSight reports this. To control such a window, start the agent from an elevated terminal.

Known limitations

  • On Windows 10, the WGC backend draws a yellow border around the captured window (Windows 11 can turn it off). That is why printwindow is tried first and WGC is used only when PrintWindow returns a black frame.

  • Minimized windows have no pixels. snapshot(restore_minimized=true) restores the window without activating it.

  • Background key presses (background=true) are reliable for single keys only: most apps read the real keyboard state for Ctrl / Alt / Shift.

  • Background clicks work for classic Win32 and WinForms. Chromium and most games ignore them; use an element id, or leave background off.

  • Chromium and Electron build their accessibility tree only after a client asks for it. snapshot retries once automatically.

Layout

src/omnisight/
  winutil.py   window enumeration, DPI awareness, elevation, display affinity
  capture.py   capture backend chain, black-frame detection, WGC session pool
  uia.py       UI Automation tree (one cache request) and actions
  ocr.py       RapidOCR / Windows OCR
  inputs.py    SendInput and background message input
  server.py    MCP tools

Development

uv sync --extra ocr --group dev
uv run pytest

License

Apache-2.0

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables Windows UI Automation screen-to-text and computer-use via MCP, allowing agents to observe, decide, and act on Windows applications through accessibility data without OCR.
      1
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables agents to control Windows desktop applications through UI Automation and Win32, supporting window management, mouse/keyboard input, element inspection, screenshots, OCR, macro recording, and autonomous multi-app missions. It also provides an HTTP/stdio MCP interface, an operator UI, and optional telemetry and safety controls.
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables MCP agents to drive Windows applications by targeting individual windows—clicking, typing, scrolling, dragging, and reading accessibility trees or screenshots—without stealing the user's focus, mouse, or keyboard. Runs entirely locally alongside the user, with per-app approval and no cloud account or usage limits.
      MIT
    • A
      license
      B
      quality
      B
      maintenance
      Enables AI agents to observe and control local browsers and desktop applications over MCP, including reading UI elements and differences, capturing images, and performing permitted clicks, typing, selections, scrolling, and multi-window operations.
      16
      2
      MIT