browserbase_stagehand_observe
Locate interactive web page elements from specific instructions to identify buttons, forms, or clickable components for automation tasks.
Instructions
Find interactive elements on the page from an instruction; optionally return an action.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| instruction | Yes | Detailed instruction for what specific elements or components to observe on the web page. This instruction must be extremely specific and descriptive. For example: 'Find the red login button in the top right corner', 'Locate the search input field with placeholder text', or 'Identify all clickable product cards on the page'. The more specific and detailed your instruction, the better the observation results will be. Avoid generic instructions like 'find buttons' or 'see elements'. Instead, describe the visual characteristics, location, text content, or functionality of the elements you want to observe. This tool is designed to help you identify interactive elements that you can later use with the act tool for performing actions like clicking, typing, or form submission. | |
| returnAction | No | Whether to return the action to perform on the element. If true, the action will be returned as a string. If false, the action will not be returned. |
Implementation Reference
- src/tools/observe.ts:34-62 (handler)The handler function that performs the observation by calling stagehand.observe with the given instruction and returns the observations as text.
async function handleObserve( context: Context, params: ObserveInput, ): Promise<ToolResult> { const action = async (): Promise<ToolActionResult> => { try { const stagehand = await context.getStagehand(); const observations = await stagehand.observe(params.instruction); return { content: [ { type: "text", text: `Observations: ${JSON.stringify(observations)}`, }, ], }; } catch (error) { const errorMsg = error instanceof Error ? error.message : String(error); throw new Error(`Failed to observe: ${errorMsg}`); } }; return { action, waitForNetwork: false, }; } - src/tools/observe.ts:13-32 (schema)Input schema using Zod for validation of the 'instruction' parameter and the tool schema defining the tool name, description, and input schema.
const ObserveInputSchema = z.object({ instruction: z.string().describe( `Detailed instruction for what specific elements or components to observe on the web page. This instruction must be extremely specific and descriptive. For example: 'Find the red login button in the top right corner', 'Locate the search input field with placeholder text', or 'Identify all clickable product cards on the page'. The more specific and detailed your instruction, the better the observation results will be. Avoid generic instructions like 'find buttons' or 'see elements'. Instead, describe the visual characteristics, location, text content, or functionality of the elements you want to observe. This tool is designed to help you identify interactive elements that you can later use with the act tool for performing actions like clicking, typing, or form submission.`, ), }); type ObserveInput = z.infer<typeof ObserveInputSchema>; const observeSchema: ToolSchema<typeof ObserveInputSchema> = { name: "browserbase_stagehand_observe", description: `Find and identify interactive elements on the page based on an instruction.`, inputSchema: ObserveInputSchema, }; - src/tools/observe.ts:64-70 (registration)Tool registration object that combines the schema and handler for the observe tool.
const observeTool: Tool<typeof ObserveInputSchema> = { capability: "core", schema: observeSchema, handle: handleObserve, }; export default observeTool; - src/tools/index.ts:19-27 (registration)Central registration of all tools including the observeTool in the exported TOOLS array.
export const TOOLS = [ ...sessionTools, navigateTool, actTool, extractTool, observeTool, screenshotTool, getUrlTool, ];