click
Click elements on web pages using numbered labels from annotated screenshots to navigate and interact with content.
Instructions
Click an element on the page specified by its numbered label from the annotated screenshot
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| label | Yes | The label of the element to click, as shown in the annotated screenshot |
Implementation Reference
- src/index.ts:647-704 (handler)The main handler function for the 'click' tool. It selects the element by its data-label attribute (set by the marking script), checks for target='_blank' links to handle new tabs by navigating directly, and performs the click or navigation.
async function handleClick(page: Page, args: any): Promise<CallToolResult> { const { label } = args; if (!label) { return { isError: true, content: [ { type: "text", text: "Label parameter is required for clicking elements", }, ], }; } const selector = `[data-label="${label}"]`; try { // Wait for the element to be visible await page.waitForSelector(selector, { visible: true }); // Evaluate if the element has a target="_blank" anchor type ClickResult = | { hasTargetBlank: true; href: string } | { hasTargetBlank: false }; const result = await page.$eval(selector, (element): ClickResult => { const anchor = element.closest("a"); if (anchor && anchor.target === "_blank" && anchor.href) { return { hasTargetBlank: true, href: anchor.href }; } return { hasTargetBlank: false }; }); // If the element navigates to a new tab, go to that href instead if (result.hasTargetBlank) { await page.goto(result.href); } else { await page.click(selector); } // Success - no error content return { isError: false, content: [{ type: "text", text: `Clicked element with label ${label}.` }], }; } catch (e) { return { isError: true, content: [ { type: "text", text: `Could not find clickable element with label ${label}. Error: ${ (e as Error).message }`, }, ], }; } } - src/index.ts:482-497 (schema)The schema definition for the 'click' tool, including name, description, and input schema specifying a required 'label' number.
{ name: "click", description: "Click an element on the page specified by its numbered label from the annotated screenshot", inputSchema: { type: "object", properties: { label: { type: "number", description: "The label of the element to click, as shown in the annotated screenshot", }, }, required: ["label"], }, }, - src/index.ts:922-923 (registration)The registration/dispatch point in the main handleToolCall function's switch statement that routes 'click' tool calls to the handleClick handler.
case "click": result = await handleClick(page, args); - src/index.ts:667-669 (schema)Type definition used within the handler for the result of evaluating if the clicked element opens in a new tab.
type ClickResult = | { hasTargetBlank: true; href: string } | { hasTargetBlank: false };