scene_describe
Converts a window into text with element names, coordinates, and a text digest, enabling reliable UI interaction when accessibility trees or visual inspection are unavailable.
Instructions
Describe a window as text: elements with names and coordinates plus a text digest. Use it when the target has no usable accessibility tree (Electron / canvas apps) or when you cannot look at a picture. Every bbox/center is window-relative physical pixels in the grid window.size describes, so a center is a valid click point inside the window; the text digest prints the window's screen origin (@x,y), so absolute = origin + center, added once. source=uia reads the accessibility tree only and never runs OCR; source=ocr reads pixels only; auto blends both. capture.verified=false means the frame could not be proven to be this window, so its OCR and OpenCV results were suppressed; a failed read carries error_code and reason (INVALID_SOURCE, INVALID_REGION, CAPTURE_FAILED, OCR_UNAVAILABLE, UIA_EMPTY, SCENE_FILTERED_EMPTY, UNVERIFIED_FRAME). Read-only: it never clicks or types, and the window may sit behind other windows.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"hwnd": "<hwnd>"
},
"name": "scene_describe"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| hwnd | Yes | ||
| source | No | uia = accessibility tree only: no OCR, no capture, no OpenCV shapes | auto |
| max_items | No | ||
| include_shapes | No | also report controls that carry no text; needs a pixel tier (auto/ocr), since source=uia reads no pixels |