Locate by description
locateFinds a described UI target in the current screenshot using a grounding model when refs and marks fail; returns pixel candidate points and an annotated screenshot when unsure.
Instructions
Find a described target in the current screenshot with a grounding model when refs and marks fail. Returns candidate points in image pixels; when not confident, up to 3 candidates and an annotated screenshot.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"description": "<description>",
"observation_id": "<observation_id>"
},
"name": "locate"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| description | Yes | What to find, e.g. 'the blue Export button' | |
| observation_id | Yes | observation_id of the observation you act on |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||