find_by_vision
Capture a screenshot and use an external vision model to locate UI elements or answer questions about the screen, returning coordinates or text.
Instructions
Take a screenshot and delegate visual analysis to an external vision model, returning the result. Use when visual analysis should be performed by a separate model rather than by the calling agent. Requires an external API key: ANTHROPIC_API_KEY (claude-), OPENAI_API_KEY (gpt-/o-series), GEMINI_API_KEY (gemini-), or AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY (amazon.nova-).
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| model | Yes | Vision model to use. Determines which credentials are required: claude-* → ANTHROPIC_API_KEY, gpt-*/o-series → OPENAI_API_KEY, gemini-* → GEMINI_API_KEY, amazon.nova-* → AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY. | |
| prompt | Yes | Question or instruction about the screenshot | |
| responseFormat | No | "coordinates" (default) locates an element and returns JSON {x,y,label} with converted screen coordinates. "text" answers a general question about the screen in plain text. | coordinates |
| includeAnnotatedImage | No | When true, also returns the numbered-badge screenshot actually sent to the vision model, as an image content block - useful for debugging why a particular tag was picked. |