capture_som
Capture a window with numbered marks on interactive controls so agents can act by mark number or exact screen rectangle instead of guessing pixels.
Instructions
Capture a window with numbered marks on every interactive element.
This is the tool to use when you need to understand *where* things are, not
just *what* they are. The accessibility tree decides the boxes, so marks are
pixel-exact and never hallucinated, and every mark in the legend carries a
real screen rect.
Prefer this over capture_window when you plan to act on the result: answer
with a mark number, or use the rect directly and skip vision entirely.
max_marks caps how many controls are numbered (dense toolbars get crowded);
the highest-value controls are kept - named ones first, then larger ones.
max_side downscales the image before drawing, so badges stay sharp.
Falls back to a plain unmarked capture with an explanation when the window
exposes no UI tree (Chromium/Electron without --force-renderer-accessibility).
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"window": "<window>"
},
"name": "capture_som"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| window | Yes | ||
| max_side | No | ||
| max_marks | No |