ui_ocr
Reads text from windows with almost no UI tree via Windows OCR, returning each line with a reference and screen rectangle so agents can target text-based click actions.
Instructions
Only when ui_tree shows almost nothing (games, Qt, Sciter, canvas UIs): the window's text read by Windows OCR, one line per text with a ref (t3) and screen rect. ui_act takes a t-ref for click, double-click, right-click, hover, wheel and drag, posted at the text's center as mouse messages (no real cursor, no focus change). About 0.5 to 2 s; works while covered. t-refs expire after 60 s.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"title": "<title>"
},
"name": "ui_ocr"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| title | Yes | part of the window title (case-insensitive; an exact title wins) or a handle from list_windows (0x...) |