Enhanced Web Scraper MCP Server

Instructions

Implementation Reference

• server.js:966-997 (handler)

  The main handler function for 'wait_for_element' tool. Launches browser, navigates to URL, uses findElement helper to wait for selector, retrieves element properties, and returns success details including wait time, visibility, text, testID, and accessibility label.

    async waitForElement(args) {
      this.validateArgs(args, ['url', 'selector']);
      const { url, selector, timeout = TIMEOUTS.DEFAULT, browser: browserType = 'chromium' } = args;
      
      const { browser, context } = await this.getBrowser(browserType);
      const page = await context.newPage();
      
      try {
        await page.goto(url, { waitUntil: 'networkidle' });
        
        const startTime = Date.now();
        const { element, usedSelector } = await this.findElement(page, selector, timeout);
        
        const waitTime = Date.now() - startTime;
        const text = await element.textContent();
        const isVisible = await element.isVisible();
        const testId = await element.getAttribute('data-testid');
        const accessibilityLabel = await element.getAttribute('aria-label');
  
        return {
          content: [{
            type: 'text',
            text: `✅ Element found: ${usedSelector}
  Wait time: ${waitTime}ms
  Visible: ${isVisible}
  Text content: "${text?.trim()}"${testId ? `\nTestID: ${testId}` : ''}${accessibilityLabel ? `\nAccessibility Label: ${accessibilityLabel}` : ''}`
          }]
        };
      } finally {
        await context.close();
        await browser.close();
      }

• server.js:447-475 (registration)

  Registration of the 'wait_for_element' tool in the list of tools returned by ListToolsRequestSchema, including name, description, and input schema.

  {
    name: 'wait_for_element',
    description: 'Wait for an element to appear on the page (useful for dynamic React content)',
    inputSchema: {
      type: 'object',
      properties: {
        url: {
          type: 'string',
          description: 'URL to monitor'
        },
        selector: {
          type: 'string',
          description: 'CSS selector to wait for'
        },
        timeout: {
          type: 'number',
          default: 10000,
          description: 'Maximum time to wait in milliseconds'
        },
        browser: {
          type: 'string',
          enum: ['chromium', 'firefox', 'webkit'],
          default: 'chromium',
          description: 'Browser engine to use'
        }
      },
      required: ['url', 'selector']
    }
  },

• server.js:451-473 (schema)

  Input schema definition for the 'wait_for_element' tool, specifying required url and selector, optional timeout and browser.

  type: 'object',
  properties: {
    url: {
      type: 'string',
      description: 'URL to monitor'
    },
    selector: {
      type: 'string',
      description: 'CSS selector to wait for'
    },
    timeout: {
      type: 'number',
      default: 10000,
      description: 'Maximum time to wait in milliseconds'
    },
    browser: {
      type: 'string',
      enum: ['chromium', 'firefox', 'webkit'],
      default: 'chromium',
      description: 'Browser engine to use'
    }
  },
  required: ['url', 'selector']

• server.js:82-98 (helper)

  Helper function used by waitForElement to locate the element using multiple selector strategies: exact CSS, data-testid, aria-label, accessibilityLabel. Tries each with proportional timeout and returns the element and used selector.

  async findElement(page, selector, timeout = TIMEOUTS.DEFAULT) {
    const selectors = [
      selector,
      `[data-testid="${selector}"]`,
      `[aria-label="${selector}"]`,
      `[accessibilityLabel="${selector}"]`
    ];
    
    for (const sel of selectors) {
      try {
        await page.waitForSelector(sel, { timeout: timeout / selectors.length });
        return { element: await page.$(sel), usedSelector: sel };
      } catch (e) {
        continue;
      }
    }
    throw new Error(`Element not found with any selector: ${selector}`);