browserbase_stagehand_observe
Locate interactive web page elements by providing specific visual or functional descriptions, enabling precise identification for subsequent automation actions.
Instructions
Find interactive elements on the page from an instruction; optionally return an action.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| instruction | Yes | Detailed instruction for what specific elements or components to observe on the web page. This instruction must be extremely specific and descriptive. For example: 'Find the red login button in the top right corner', 'Locate the search input field with placeholder text', or 'Identify all clickable product cards on the page'. The more specific and detailed your instruction, the better the observation results will be. Avoid generic instructions like 'find buttons' or 'see elements'. Instead, describe the visual characteristics, location, text content, or functionality of the elements you want to observe. This tool is designed to help you identify interactive elements that you can later use with the act tool for performing actions like clicking, typing, or form submission. |
Implementation Reference
- src/tools/observe.ts:35-63 (handler)The handleObserve function that executes the tool's logic: retrieves Stagehand from context, performs observation based on the instruction, and returns the observations as JSON string in text content.
async function handleObserve( context: Context, params: ObserveInput, ): Promise<ToolResult> { const action = async (): Promise<ToolActionResult> => { try { const stagehand = await context.getStagehand(); const observations = await stagehand.observe(params.instruction); return { content: [ { type: "text", text: `Observations: ${JSON.stringify(observations)}`, }, ], }; } catch (error) { const errorMsg = error instanceof Error ? error.message : String(error); throw new Error(`Failed to observe: ${errorMsg}`); } }; return { action, waitForNetwork: false, }; } - src/tools/observe.ts:29-33 (schema)The tool schema defining the name 'browserbase_stagehand_observe', description, and input schema (ObserveInputSchema defined earlier with 'instruction' field).
const observeSchema: ToolSchema<typeof ObserveInputSchema> = { name: "browserbase_stagehand_observe", description: `Find interactive elements on the page from an instruction; optionally return an action.`, inputSchema: ObserveInputSchema, }; - src/tools/observe.ts:65-71 (registration)Defines the complete observeTool object combining schema and handler, exported for use in tools index and MCP server registration.
const observeTool: Tool<typeof ObserveInputSchema> = { capability: "core", schema: observeSchema, handle: handleObserve, }; export default observeTool; - src/tools/index.ts:21-30 (registration)Registers the observeTool (imported from ./observe.js) in the TOOLS array used by the MCP server.
export const TOOLS = [ ...sessionTools, navigateTool, actTool, extractTool, observeTool, screenshotTool, getUrlTool, agentTool, ]; - src/index.ts:168-198 (registration)The MCP server loop that registers all tools from TOOLS, including browserbase_stagehand_observe, by calling server.tool with the schema name and a wrapper around context.run.
const tools: MCPToolsArray = [...TOOLS]; // Register each tool with the Smithery server tools.forEach((tool) => { if (tool.schema.inputSchema instanceof z.ZodObject) { server.tool( tool.schema.name, tool.schema.description, tool.schema.inputSchema.shape, async (params: z.infer<typeof tool.schema.inputSchema>) => { try { const result = await context.run(tool, params); return result; } catch (error) { const errorMessage = error instanceof Error ? error.message : String(error); process.stderr.write( `[Smithery Error] ${new Date().toISOString()} Error running tool ${tool.schema.name}: ${errorMessage}\n`, ); throw new Error( `Failed to run tool '${tool.schema.name}': ${errorMessage}`, ); } }, ); } else { console.warn( `Tool "${tool.schema.name}" has an input schema that is not a ZodObject. Schema type: ${tool.schema.inputSchema.constructor.name}`, ); } });