Skip to main content
Glama
yangpeng366

OpenEyes

by yangpeng366
OverviewSchemaRelated ServersScoreDiscussions
Python
Local

OpenEyes（开源点睛）

AI-friendly computer-use primitives for Windows / macOS / Linux. Capture. Detect. Click. Let any LLM agent drive any desktop GUI.

License: MIT Python 3.10+ Windows / macOS / Linux

GitHub: https://github.com/yangpeng366/openeyes · 中文 · English · Quick start · Architecture · Roadmap · License

中文

OpenEyes（点睛） 是一个开源的 AI 友好电脑使用平台。基于「see → resolve → act」三段式原语，让任何 LLM agent 都能可靠地操控桌面 GUI 应用；提供 Vimium 风格字母 hint overlay 作为人类可调试的回退通道，并通过 MCP server 暴露给所有 agent。

为什么做这个

现状

痛点

Neverclick

闭源付费，仅 Windows，无 API

OmniParser / OS-Atlas

只做 detection，没有 actuation，模型重

Anthropic Computer Use / OpenAI Operator

仅 Linux VM，看不见本机，不可定制

UiBot / APA

录制型，对动态 UI 适应性差

pywinauto / xdotool

纯库，无 AI 友好的 element schema

OpenEyes 把这些能力合成一个 MIT 开源、跨平台、AI 友好的统一平台。

特性

  • AI 友好 element schema：每个 UI 元素输出结构化 JSON（bbox / center / control_type / name / automation_id / class / state）

  • 易点击：UIA 命中 → 自动点中心；不命中 → vision bbox → 点中心；都不命中 → 字母 hint overlay

  • 安全：默认 dry-run，所有状态改变操作可审计

  • MCP 优先：所有原语通过 MCP server 暴露给任何 agent

  • 可插拔 vision backend：OmniParser / Florence-2 / 自训练

  • 跨平台：Windows（首发）/ macOS / Linux，统一抽象

设计文档

完整设计书：https://my.feishu.cn/docx/Ul6gdMULGo5VfzxDVAYcVoX6n9e （中文，含架构图 / 模块清单 / 路线图）

Related MCP server: helix-pilot

English

OpenEyes is an open-source, AI-friendly computer-use platform. Built on a "see → resolve → act" pipeline, it lets any LLM agent reliably drive any desktop GUI. Includes a Vimium-style letter-hint overlay as a human-debuggable fallback, and exposes every primitive via MCP server for any agent.

Why

Today

Pain

Neverclick

Closed-source, paid, Windows-only, no API

OmniParser / OS-Atlas

Detection only, no actuation, heavy models

Anthropic Computer Use / Operator

Linux VMs only, opaque, no audit

UiBot / APA

Record/playback, weak on dynamic UIs

pywinauto / xdotool

Raw libs, no AI-friendly schema

OpenEyes fuses these into one MIT-licensed, cross-platform, AI-friendly platform.

Features

  • AI-friendly element schema — every interactive element exposes structured JSON

  • Easy click — UIA hit → bbox center; fallback vision bbox → center; final fallback letter hint

  • Safe by default — dry-run; full audit trail for state-changing actions

  • MCP-first — every primitive exposed as an MCP tool to any agent

  • Pluggable vision backend — OmniParser / Florence-2 / custom

  • Cross-platform — Windows (first), macOS, Linux, with one unified abstraction

Quick start

Windows

git clone https://github.com/yangpeng366/openeyes.git
cd openeyes
pip install -e ".[windows,mcp]"

# 1. list visible top-level windows
eyes windows list

# 2. capture a window or the full screen
eyes capture --window 123456 --out shot.png

# 3. enumerate interactive elements
eyes detect --window 123456 --pretty

# 4. click by text (resolves to bbox center)
eyes click --window 123456 --name-contains "Submit" --dry-run
eyes click --window 123456 --name-contains "Submit" --go

# 5. start MCP server (for Codex / Claude / Cursor)
eyes-mcp

First showcase — 飞书 client

See examples/feishu_first_test.py.

python examples\feishu_first_test.py --dry-run

Architecture

┌────────────────────────────────────────────────────────────────┐
│  L5  Orchestration    Codex / Claude / Cursor / 自定义 Agent     │
├────────────────────────────────────────────────────────────────┤
│  L4  AI Layer         Intent → Element (LLM 驱动 + 缓存)        │
├────────────────────────────────────────────────────────────────┤
│  L3  Resolver         selector → coord   (UIA / Vision / Hint)  │
├────────────────────────────────────────────────────────────────┤
│  L2  Perceive         capture + detect    (UIA / AX / AT-SPI)    │
├────────────────────────────────────────────────────────────────┤
│  L1  Actuate          mouse / key / drag  (Win32 / CG / XTest)  │
├────────────────────────────────────────────────────────────────┤
│  L0  Platform         Windows / macOS / Linux                   │
└────────────────────────────────────────────────────────────────┘

See docs/architecture.md for the full design.

Roadmap

  • v0.1.0 — MVP: UIA capture/detect/click + CLI + MCP + 飞书 first showcase

  • v0.2.0 — Vision backend (OmniParser v2 / Florence-2)

  • v0.3.0 — macOS (AXUIElement) + Linux (AT-SPI)

  • v0.4.0 — Vimium-style letter hint overlay

  • v0.5.0 — LLM intent resolver

  • v0.6.0 — Audit log + replay

  • v1.0.0 — Rust hot path + cross-platform binaries

Repository layout

openeyes/
├── openeyes/              # main package
│   ├── core/              # windows + capture + detect + click primitives
│   │   ├── windows.py     # EnumWindows wrapper
│   │   ├── capture.py     # PIL.ImageGrab wrapper
│   │   ├── schema.py      # Element / WindowInfo dataclasses
│   │   ├── selector.py    # element find / filter
│   │   └── _platform.py   # cross-platform backend selection
│   ├── backends/
│   │   └── uia.py         # Windows UIA backend (pywinauto)
│   ├── actuators/
│   │   └── win32.py       # Windows mouse/keyboard input
│   ├── cli/               # `eyes` command
│   │   └── main.py
│   └── mcp/               # MCP server
│       └── server.py
├── examples/
│   ├── feishu_first_test.py
│   └── anyvpn_keepalive.py
├── tests/
│   └── test_smoke.py
├── .codex-plugin/
│   └── plugin.json
├── skills/openeyes/
│   └── SKILL.md
├── pyproject.toml
├── LICENSE                 # MIT
└── README.md

License

MIT — see LICENSE.

This server cannot be installed

A
license - permissive license
-
quality - not tested
B
maintenance

How are these scores calculated?

Maintenance

–Maintainers
–Response time
–Release cycle
–Releases (12mo)
Commit activity

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Servers

  • helix-pilot

    tsunamayo7
    A
    license
    -
    quality
    C
    maintenance
    GUI automation MCP server that enables AI agents to see and control the Windows desktop using a local Vision LLM (Ollama), supporting screenshot analysis, mouse/keyboard actions, and autonomous task execution.
    4
    MIT

  • openowl

    mihir-kanzariya
    A
    license
    -
    quality
    B
    maintenance
    An MCP server that gives any AI assistant eyes and hands on your desktop — screenshots, clicking, typing, OCR, window management, accessibility-tree queries, workflow recording.
    5
    Apache 2.0

  • ControlMCP

    nix18
    A
    license
    -
    quality
    D
    maintenance
    An MCP server that enables LLMs to see and control a computer — screen capture, window management, mouse and keyboard automation — with a structured plan-execute workflow for complex desktop automation.
    GPL 3.0

View all related MCP servers

Related MCP Connectors

  • Kody

    Personal assistant MCP server with search, execute, packages, jobs, secrets, and integrations.

  • mcp-aichat

    MCP server for AI dialogue using various LLM models via AceDataCloud

  • hithereiamaliff-mcp-nextcloud

    A comprehensive Model Context Protocol (MCP) server that enables AI assistants to interact with yo…

View all MCP Connectors

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/yangpeng366/openeyes'

If you have feedback or need assistance with the MCP directory API, please join our Discord server