ui_locate
Locates UI controls such as buttons, inputs, or icons by matching target descriptions to OCR text, returning pixel coordinates for clicking or typing.
Instructions
UI 元素定位(文本锚定优先):目标描述 → 关键词 → OCR 文本匹配 → 控件框(像素级)。按钮/输入框/图标等 UI 点击类目标的备选精定位;返回 matched + 候选列表供 VLM 确认。
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"image": "<image>",
"target": "<target>"
},
"name": "ui_locate"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| type | No | 目标类型过滤,可选 | |
| image | Yes | ||
| coords | No | 返回坐标单位:pixel(默认)或 norm(0-1000 归一化) | |
| target | Yes | 目标描述,如:点击「登录」按钮 / 搜索框输入 hello / 右上角图标 |