Skip to main content
Glama

EDM — Everything Desktop on Me

High-resolution Windows desktop vision, region snipping, zoom, UIA/OCR perception, and full mouse/keyboard control for AI agents via MCP and CLI.

Status: Planning & architecture complete (v0.1 specification ready). Source implementation begins at Phase 0.
Türkçe: README.tr.md · Design & Plan: docs/plan/00-TASARIM.md


Why EDM Exists

AI agents trying to drive real Windows desktop apps (AutoCAD, WinRAR, Chrome dialogs, AOC G-Menu, etc.) repeatedly fail on the first attempt and loop because:

  1. Fragile window binding: Window-scoped capture (WGC / PrintWindow) times out or returns blank frames on DirectX/OpenGL viewports, splash screens, popups, or system tray menus.

  2. UI Automation blind spots: Standard controls expose accessibility trees, while CAD viewports and custom hardware UIs appear as opaque black boxes.

  3. DPI & coordinate drift: Scaled monitors (125% / 150% / 4K) cause pixel guesses to miss small toolbar icons.

  4. Silent OS blocks (UIPI & isolated desktops): Elevated apps silently drop standard SendInput, and sandboxed agent shells run on an isolated desktop (WinSta0\exebox-...) instead of the user's visible WinSta0\Default session.

EDM solves this with an observation-bound, PrintScreen-first, closed-loop architecture:

  • See everything actually on screen: Virtual screen, specific monitor, visible window crop (GDI composite), or rectangular region — with WGC and PrintWindow fallbacks for occluded windows and blank-frame detection.

  • Coarse-to-fine zoom: Inspect full-screen layout, then crop/zoom any dense ribbon or tiny button at native pixel resolution (preserving lossless PNG on small CAD crops) while EDM automatically maps coordinates back to the physical virtual screen (pixel or norm1000).

  • Human-like full input & drawing control: Single/double/right/middle clicks, straight drags, smooth multi-point continuous strokes (draw_path for Paint/CAD curves), held buttons/keys (mouse_button, key_state with automatic release-on-error), Unicode typing, shortcuts, and validated multi-step sequences.

  • Closed-loop safety: Per-Monitor-V2 DPI lock, UIPI elevation detection (ELEVATION_REQUIRED), stale-observation guard (window geometry + 24×24 pixel patch verification), inter-agent input lock, and instant global kill switch (Ctrl+Alt+Shift+F12 + (0,0) corner).

  • Multi-client ready: One core engine (src/edm) exposed as both an MCP server (stdio + user-started loopback HTTP) and a JSON CLI (edm), designed for Antigravity, Codex Desktop, Claude Code, Gemini CLI (agy), and local scripts.


Related MCP server: com-use-mcp

Planned Architecture (v0.1)

AI Clients (Antigravity / Codex / Claude Code / Gemini CLI)  ──MCP──▶  mcp_server.py / http_server.py
User / Local Scripts                                        ──CLI──▶  cli.py
                                                                          │
                                                                    service.Edm (single facade)
                                                                          │
    ┌──────────────────────┬──────────────────────┬───────────────────────┴──────────────────────┐
    ▼                      ▼                      ▼                                              ▼
capture/                observe/              perception/                                    actions/
gdi → wgc → pw         store, render, diff    uia (SoM), winrt ocr (tr-TR)                  guard, killswitch, input

Key design & planning documents (in Turkish):


Security & Privacy Model

  • Local-only: No cloud telemetry or outbound network calls. Screenshots and action logs stay strictly under %LOCALAPPDATA%\EDM\ (auto-pruned at 7 days / 2 GB) and are never committed to Git.

  • Explicit boundaries: EDM never switches Windows desktops, patches sandboxes, or bypasses UAC. If an elevated app (e.g., G-Menu) requires admin integrity, EDM detects it and asks the user to launch EDM elevated if desired.

  • Screenshot sensitivity: Anything visible on your screen when a tool runs can be sent as tool output to whichever AI client invoked EDM. Do not leave secrets or banking sessions uncovered during agent runs.


Acknowledgments & Prior Art

EDM is written cleanly from scratch for Windows, informed by lessons from open-source GUI agents and local Windows automation work; see THIRD_PARTY_NOTICES.md.

License

MIT © 2026 Hüseyin Günaydın

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables AI assistants to autonomously control Windows 11 and 10 desktops via sub-10ms screen capture, native UI Automation element inspection, and zero-lag keyboard and mouse input. Combines a visual plane with a semantic plane and stall detection so agents can operate real applications reliably without vision-only guessing.
      262 npm
      171 PyPI
      7
      MIT
    • A
      license
      A
      quality
      B
      maintenance
      Enables agents to perceive and control a Windows desktop, a connected local Chromium browser, or a cloud Playwright browser through human-like mouse moves, clicks, drags, scrolling, and screenshots, forming an observe-locate-act computer-use loop. Includes safety guards such as random pauses, emergency stop, and approval prompts for destructive actions.
      11
      MIT