locate
Find an on-screen element from its description when the accessibility tree and OCR fail. Uses a vision-language model to return its bounding box.
Instructions
Locate a described element via the VLM grounding layer (read-only).
The THIRD perception layer (tree -> OCR -> VLM): when the accessibility tree and OCR cannot resolve a target on a tree-failed screen (Flutter/ Unity/games), this asks a vision-language model for the element's bounding box on the current screenshot.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| device | No | ADB serial; omit with a single connected device. | |
| target | Yes | {"by": "text"|"desc", "value": "<description>"} — describe the element you are looking for (e.g. "the blue download button"). |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||