# mcp-winaccess-win

**Version 1.8.0** · Windows-only desktop automation MCP server. Built directly on the OS APIs —
**ctypes** (`SendInput`, `ImageGrab`, Win32 windows, clipboard) and **UI Automation** via
`comtypes` — with **no `pyautogui` / `pywinauto` / `pywin32`**.

## Requirements

- **Windows 10/11 (x64)** — this package only runs on Windows.
- Python 3.10+.
- The server must run **inside an interactive desktop session** (a logged-in user), not as a
  background service — otherwise `SendInput`/UI Automation cannot reach the desktop.
- No administrator rights are required for normal user applications (see Limitations).
- OCR auto-install needs network access and 7-Zip (or an existing Tesseract install; see Troubleshooting).

## Install

```
pip install mcp-winaccess-win              # OCR support (pytesseract) is included
pip install "mcp-winaccess-win[vision]"    # optional: OpenCV template matching
```

The **Tesseract binary** (a system program, not a Python package) is downloaded and installed
automatically on first OCR use into `%LOCALAPPDATA%\mcp-winaccess\tesseract` (with `eng` + `rus`
language data). Use `--tesseract_cmd <path>` to point at an existing install, or
`--no-auto-tesseract` to disable the automatic install (see Troubleshooting).

## Run

```
python -m mcp_winaccess_win.server
# or the console script
mcp-winaccess-win
```

### Command-line arguments

| Argument | Default | Meaning |
|---|---|---|
| `--transport` | `local` | `local` (stdio) or `remote` (streamable-http) |
| `--listen` | `0.0.0.0` | HTTP bind address (remote only) |
| `--port` | `8765` | HTTP port (remote only) |
| `--token` | *(none)* | Bearer token required for HTTP (remote only) |
| `--tesseract_cmd` | `auto` | `auto`, or an explicit path to `tesseract.exe` |
| `--auto-tesseract` | on | automatic Tesseract install; `--no-auto-tesseract` disables it |
| `--no-shell` | off | disables the `run_command` tool (shell execution); enabled by default |

Local (stdio) is the default and needs no arguments. To serve over HTTP, protect the desktop
with a token — a token is **required** whenever `--listen` is not loopback:

```
mcp-winaccess-win --transport remote --listen 127.0.0.1 --port 8765 --token mysecret
```

## OpenCode config (`opencode.jsonc`)

Local (stdio) — OpenCode launches the server itself:

```jsonc
{
  "mcp": {
    "winaccess": {
      "type": "local",
      "command": ["uvx", "mcp-winaccess-win", "--tesseract_cmd", "auto"],
      "enabled": true
    }
  }
}
```

Remote (streamable-http) — the token goes in the `Authorization` header:

```jsonc
{
  "mcp": {
    "winaccess": {
      "type": "remote",
      "url": "http://127.0.0.1:8765/mcp",
      "headers": { "Authorization": "Bearer mysecret" },
      "enabled": true
    }
  }
}
```

## Architecture

| Layer | Implementation |
|---|---|
| Mouse / keyboard | `SendInput` (ctypes); text via `KEYEVENTF_UNICODE` (Unicode without the clipboard) |
| Screenshots | Pillow `ImageGrab` (`bbox`, `all_screens`, window handle) |
| Windows | ctypes `EnumWindows` / `MoveWindow` / `SetWindowPos` / `PostMessage` |
| Clipboard | Win32 clipboard via ctypes |
| UI tree, menus, dialogs | UI Automation COM via `comtypes` |
| Vision / OCR | OpenCV template matching / pytesseract with an auto-installed Tesseract binary |

Modules: `mcp_winaccess_win/server.py` (tool wrappers), `mcp_winaccess_win/adapter/windows.py`
(adapter), `mcp_winaccess_win/adapter/win32_input.py`,
`mcp_winaccess_win/adapter/win32_screen.py`, `mcp_winaccess_win/adapter/win32_window.py`,
`mcp_winaccess_win/adapter/win32_clipboard.py`,
`mcp_winaccess_win/adapter/win32_overlay.py` (element highlight), `mcp_winaccess_win/adapter/win32_notifications.py`,
`mcp_winaccess_win/adapter/win32_process.py` (launch/kill/enumerate processes),
`mcp_winaccess_win/adapter/tesseract_setup.py` (auto-install OCR engine), `mcp_winaccess_win/adapter/uia.py` (UI Automation),
`mcp_winaccess_win/adapter/base.py` (interface + shared helpers).

## Conventions

- `value` (window identifier): a partial title/class (`str`), a window handle (`int`), or a
  prefix `title:`, `class:`, `pid:N`, `exe:name.exe`.
- `control_identifier`: the `ID` returned by `get_all_controls` (`element_N`) or a control
  `Name`/`AutoID`.
- All coordinates are **absolute screen pixels** (virtual-desktop space).
- Any failure returns a string starting with `ERROR:`; success messages are human-readable.
- Tool schemas carry per-parameter descriptions, `Literal` enums for fixed choices, a human-readable
  `title`, and behavioural hints (`readOnlyHint`/`destructiveHint`/`idempotentHint`) so agents can tell
  observing tools from state-changing ones.

## Tools

94 tools. They are registered only when the adapter advertises the matching **capability**;
unsupported tools are hidden.

### Screenshots and display — `screenshot`, `screenshot_region`, `screenshot_monitor`, `screenshot_window`, `screenshot_element`
- `screenshot(grid=False, include_cursor=False)` — the whole virtual desktop (all monitors); `grid=True` overlays labeled 100 px lines; `include_cursor=True` draws the mouse pointer.
- `screenshot_jpg(path="")` — save a JPEG (temp file when `path` is empty).
- `screenshot_region(left, top, width, height, grid=False)`.
- `screenshot_monitor(index=0, grid=False)` — a single monitor.
- `screenshot_window(value, grid=False)` — capture a window (even if partially occluded).
- `screenshot_element(value, control_identifier, grid=False)` — capture a single control.
- `compare_screenshots(image_a, image_b, save_diff="")` — differing pixels/percentage between two image files.
- `list_monitors()` (includes DPI `scale`), `image_to_screen_coords(x, y, monitor_index=0)`, `get_pixel_color(x, y)`, `wait_for_pixel_color(x, y, color, timeout=10, tolerance=0)`.

### Mouse and keyboard — `input`
- `get_mouse_position()`, `move_mouse(x, y)`, `mouse_move_relative(dx, dy)`.
- `click(x=None, y=None, button="left", clicks=1)` — omit `x`/`y` to click at the current position; `clicks=2` is a double-click, `button="right"` a right-click.
- `mouse_down(x=0, y=0, button="left")` / `mouse_up(button="left")` — hold/release for manual drags.
- `drag(x_from, y_from, x_to, y_to, duration=0.5)`, `scroll(direction="down", amount=3, x=0, y=0)`.
- `type_text(text)` (Unicode), `hotkey(keys)` (e.g. `"ctrl+shift+s"`).
- `press_key(key)` — a key (`enter`, `tab`, `esc`, `f5`, letters/digits) or a system key (`win`, `volumeup`, `volumedown`, `volumemute`, `playpause`, `nexttrack`, `prevtrack`, `printscreen`).
- `key_down(key)` / `key_up(key)` — hold/release keys (e.g. Shift range selection).
- `sleep(seconds)` — wait a fixed amount of time.

### Clipboard — `clipboard`
- `clipboard_get()`, `clipboard_set(text)`, `clipboard_clear()`.
- `clipboard_set_files(paths)` / `clipboard_get_files()` — file paths (CF_HDROP).
- `clipboard_set_image(path)` / `clipboard_get_image(save_path="")` — image (CF_DIB).

### Vision / OCR — `vision`, `ocr`
- `locate_image_on_screen(image_path, confidence=0.9)`, `wait_for_image(...)`, `click_image(...)` — require the `vision` extra.
- `ocr_screen(left, top, width, height, lang="")`, `find_text_on_screen(text, confidence, lang)`, `click_text(text, confidence, lang)` — `lang` e.g. `"eng"` or `"rus+eng"`; the Tesseract binary auto-installs on first use.
- `wait_for_text(text, timeout, lang, confidence)` — waits until OCR finds text (for canvas/custom UI).

### Windows — `window_list`, `window_activate`, `window_close`, `window_minimize`, `window_maximize`, `window_move`, `window_resize`, `window_snap`, `window_topmost`
- `list_windows(process="")`, `find_window(value)`, `get_active_window()`, `get_window_state(value)` (rect, handle, pid, exe, state, monitor).
- `switch_to_window(value)`, `wait_for_window(value, timeout, require_ready=False)` (`require_ready=True` waits until visible and responsive), `wait_for_window_gone(value, timeout)`.
- `minimize_window(value)`, `maximize_window(value)`, `restore_window(value)`, `close_window(value)`.
- `move_window(value, x, y)`, `resize_window(value, x, y, width, height)`, `snap_window(value, position)`.
- `set_always_on_top(value, enabled)`.

`value` accepts a partial window title (`str`), a window handle (`int`), or a prefix: `title:`, `class:`, `pid:N`, `exe:name.exe`. `list_windows(process=...)` filters by exe name or PID.

### Menus and dialogs — `menus`, `dialogs`, `tray`
- `menu_select(value, path)` (e.g. `"File->Save As"`), `get_menu_items(value, menu="")` (discover the menu bar / a menu's items), `context_menu_click(x, y, item)`.
- `list_dialogs()`, `handle_dialog(button_text, title="")`.
- `file_dialog_set_path(path, title="")`, `file_dialog_confirm(title="")`.
- `tray_icon_click(name, action="left")`.

### UI tree (semantic controls) — `ui_tree`
- `get_all_controls(value, limit=150, query="", control_type="")` — use the returned `ID` (`element_N`)
  or `Name`/`AutoID` as `control_identifier`; each row includes `Rect` and `Center`.
- `click_element(value, control_identifier, clicks=1)` (`clicks=2` = double-click), `get_text`,
  `set_text(value, control_identifier, text, mode="value", clear=False)`
  (`mode="type"` forces focus + typing; `clear=True` selects-all before typing), `get_window_text(value)` (flat text dump),
  `select_item`,
  `toggle_checkbox`, `get_control_state`, `set_slider`, `get_selected_text`, `scroll_into_view`,
  `wait_for_element`, `drag_element`.

### Element inspection — `ui_tree`
- `get_element_at_point(x, y)` — element under a screen point.
- `get_active_element()` — the currently focused control.
- `highlight_element(value, control_identifier, duration=1.0, color="red", width=3)` — temporary rectangle (visual confirmation).
- `highlight_region(x, y, width, height, duration=1.0, color="red", border_width=3)` — temporary rectangle over an arbitrary screen region.
- `wait_for_element_gone(value, control_identifier, timeout=10)` — wait until a control disappears.
- `expand_element(value, control_identifier)` / `collapse_element(value, control_identifier)` — tree/expander controls.
- `get_control_state(value, control_identifier)` — enabled/offscreen/checked/value (works for checkboxes, menu items, sliders).
- `drag_element_to_element(from_value, from_control, to_value, to_control, duration=0.5)` — drag-and-drop between controls.
- `scroll_element(value, control_identifier, direction="down", amount=3)`.

### Lists and tables — `ui_tree`
- `get_list_items(value, control_identifier, limit=200)` — item names of a list/tree.
- `get_table_data(value, control_identifier, limit=200)` — table/grid rows (cells joined by ` | `).

### Notifications — `notifications`
- `list_notifications()` — current Windows notifications via the `UserNotificationListener` API (no UI opened).
- `dismiss_notifications()` — clears all notifications (UI clear-all button; the WinRT clear API is unavailable to desktop apps).

### Virtual desktops — `desktop`
- `switch_desktop(direction="right")` — switch to the adjacent desktop.
- `move_window_to_desktop(value, direction="right")` — move a window to the adjacent desktop.

### Processes and shell — `process`, `shell`
- `run_app(path, args="", cwd="")` — launch an application detached (returns its PID); pair with `wait_for_window(require_ready=True)`.
- `kill_process(pid_or_name)` — force-kill a process tree (irreversible; prefer `close_window`).
- `list_processes(filter="")` — list running processes as `PID N | name.exe`.
- `run_command(command, cwd="", timeout=30, shell="cmd")` — run a console command and return its exit code and output (`shell="powershell"` for PowerShell). Disable with `--no-shell`.

## Examples

```
# 12 + 34 in Calculator
click_element("Калькулятор", "clearButton")
click_element("Калькулятор", "num1Button")
click_element("Калькулятор", "num2Button")
click_element("Калькулятор", "plusButton")
click_element("Калькулятор", "num3Button")
click_element("Калькулятор", "num4Button")
click_element("Калькулятор", "equalButton")
get_text("Калькулятор", "CalculatorResults")   # -> 46

# Screenshot with a coordinate grid
screenshot(grid=True)
```

## Limitations

- **UIPI**: `SendInput` cannot drive windows with higher integrity (elevated apps) from a
  non-elevated process. Run the server elevated to automate those.
- **Foreground activation**: `switch_to_window` uses the Alt-unlock + `SetForegroundWindow`
  technique; some always-on-top/system windows may still refuse focus.
- **UWP apps** (e.g. Calculator) can expose two `ApplicationFrameWindow` instances (one minimized);
  the adapter prefers the visible/largest frame.
- **DPI**: the process is per-monitor-v2 DPI aware, so coordinates match physical pixels.

## Troubleshooting

- **OCR fails** — the Tesseract binary is auto-installed on first use (downloaded via the UB-Mannheim
  installer, extracted with 7-Zip, into `%LOCALAPPDATA%\mcp-winaccess\tesseract`). If that fails
  (no network / no 7-Zip), install Tesseract manually and pass `--tesseract_cmd <path>`. Language
  data `eng` and `rus` are fetched automatically; add more by dropping `*.traineddata` into the
  managed `tessdata`.
- **Vision tools missing** — install the `vision` extra (`opencv-python`).
- **UI tree tools missing** — `comtypes` is required (installed with the package).

## Tests

The `tests/` suite was removed; verification is done manually, function by function, against a
live desktop session.