webscraping_ai_html

Implementation Reference

• src/index.js:265-283 (registration)

  Registration of the 'webscraping_ai_html' tool via server.tool() with its schema definition (url, return_script_result, format, and common options) and the handler function.

  server.tool(
    'webscraping_ai_html',
    {
      url: z.string().describe('URL of the target page.'),
      return_script_result: z.boolean().optional().describe('Return result of the custom JavaScript code execution.'),
      format: z.enum(['json', 'text']).optional().describe('Response format (json or text).'),
      ...commonOptionsSchema
    },
    async ({ url, return_script_result, format, ...options }) => {
      try {
        const result = await client.html(url, { ...options, return_script_result });
        const content = format === 'json' ? JSON.stringify({ html: result }) : result;
        return createSanitizedResponse(content, url);
      } catch (error) {
        const errorObj = JSON.parse(error.message);
        return createSanitizedResponse(JSON.stringify(errorObj), url, true);
      }
    }
  );

• src/index.js:267-272 (schema)

  Input schema for the 'webscraping_ai_html' tool: url (string), return_script_result (optional boolean), format (optional enum of 'json' or 'text'), and spread commonOptionsSchema.

  {
    url: z.string().describe('URL of the target page.'),
    return_script_result: z.boolean().optional().describe('Return result of the custom JavaScript code execution.'),
    format: z.enum(['json', 'text']).optional().describe('Response format (json or text).'),
    ...commonOptionsSchema
  },

• src/index.js:273-282 (handler)

  Handler function for 'webscraping_ai_html': calls client.html(url, options) and formats the result based on the 'format' parameter (JSON with html key or raw text).

  async ({ url, return_script_result, format, ...options }) => {
    try {
      const result = await client.html(url, { ...options, return_script_result });
      const content = format === 'json' ? JSON.stringify({ html: result }) : result;
      return createSanitizedResponse(content, url);
    } catch (error) {
      const errorObj = JSON.parse(error.message);
      return createSanitizedResponse(JSON.stringify(errorObj), url, true);
    }
  }

• src/index.js:90-95 (helper)

  The client.html() helper method that makes the actual API request to the '/html' endpoint with the URL and options.

  async html(url, options = {}) {
    return this.request('/html', {
      url,
      ...options
    });
  }

• src/index.js:216-228 (registration)

  The commonOptionsSchema shared across all tools, defining timeout, js, js_timeout, wait_for, proxy, country, custom_proxy, device, error_on_404, error_on_redirect, and js_script.

  const commonOptionsSchema = {
    timeout: z.number().optional().default(DEFAULT_TIMEOUT).describe(`Maximum web page retrieval time in ms (${DEFAULT_TIMEOUT} by default, maximum is 30000).`),
    js: z.boolean().optional().default(DEFAULT_JS_RENDERING).describe(`Execute on-page JavaScript using a headless browser (${DEFAULT_JS_RENDERING} by default).`),
    js_timeout: z.number().optional().default(DEFAULT_JS_TIMEOUT).describe(`Maximum JavaScript rendering time in ms (${DEFAULT_JS_TIMEOUT} by default).`),
    wait_for: z.string().optional().describe('CSS selector to wait for before returning the page content.'),
    proxy: z.enum(['datacenter', 'residential']).optional().default(DEFAULT_PROXY_TYPE).describe(`Type of proxy, datacenter or residential (${DEFAULT_PROXY_TYPE} by default).`),
    country: z.enum(['us', 'gb', 'de', 'it', 'fr', 'ca', 'es', 'ru', 'jp', 'kr', 'in']).optional().describe('Country of the proxy to use (US by default).'),
    custom_proxy: z.string().optional().describe('Your own proxy URL in "http://user:password@host:port" format.'),
    device: z.enum(['desktop', 'mobile', 'tablet']).optional().describe('Type of device emulation.'),
    error_on_404: z.boolean().optional().describe('Return error on 404 HTTP status on the target page (false by default).'),
    error_on_redirect: z.boolean().optional().describe('Return error on redirect on the target page (false by default).'),
    js_script: z.string().optional().describe('Custom JavaScript code to execute on the target page.')
  };