scrape_webpage_js_proxy
Scrape web pages using JavaScript execution and proxy integration for dynamic content extraction, ideal for SEO analysis and data collection.
Instructions
Scrape a web page with JS and proxy
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| country | Yes | The country to use for the proxy | |
| js_script | Yes | The javascript code to execute on the page | |
| url | Yes | The url to scrape |
Implementation Reference
- index.js:654-656 (handler)The handler logic for the 'scrape_webpage_js_proxy' tool. It destructures the input arguments (renaming url and js_script for the request body), constructs query parameters from remaining args, and makes a POST request to the '/api/v1/scrape_js_with_proxy' endpoint using the shared makeRequest method.
case 'scrape_webpage_js_proxy': const { url: proxyUrl, country, js_script: proxyScript, ...proxyParams } = args; return await this.makeRequest('/api/v1/scrape_js_with_proxy', 'POST', { url: proxyUrl, country, ...proxyParams }, { url: proxyUrl, js_script: proxyScript }, token); - index.js:305-321 (schema)Input schema definition for the 'scrape_webpage_js_proxy' tool, specifying properties for url, country, js_script, and required fields.
inputSchema: { type: 'object', properties: { url: { type: 'string', description: 'The url to scrape', }, country: { type: 'string', description: 'The country to use for the proxy', }, js_script: { type: 'string', description: 'The javascript code to execute on the page', }, }, required: ['url', 'country', 'js_script'], - index.js:302-323 (registration)Registration of the 'scrape_webpage_js_proxy' tool in the listTools response, including name, description, and input schema.
{ name: 'scrape_webpage_js_proxy', description: 'Scrape a web page with JS and proxy', inputSchema: { type: 'object', properties: { url: { type: 'string', description: 'The url to scrape', }, country: { type: 'string', description: 'The country to use for the proxy', }, js_script: { type: 'string', description: 'The javascript code to execute on the page', }, }, required: ['url', 'country', 'js_script'], }, }, - index.js:565-613 (helper)Shared helper method used by all tools, including scrape_webpage_js_proxy, to make authenticated HTTP requests to the FetchSERP API.
async makeRequest(endpoint, method = 'GET', params = {}, body = null, token = null) { const fetchserpToken = token || process.env.FETCHSERP_API_TOKEN; if (!fetchserpToken) { throw new McpError( ErrorCode.InvalidRequest, 'FETCHSERP_API_TOKEN is required' ); } const url = new URL(`${API_BASE_URL}${endpoint}`); // Add query parameters for GET requests if (method === 'GET' && Object.keys(params).length > 0) { Object.entries(params).forEach(([key, value]) => { if (value !== undefined && value !== null) { if (Array.isArray(value)) { value.forEach(v => url.searchParams.append(`${key}[]`, v)); } else { url.searchParams.append(key, value.toString()); } } }); } const fetchOptions = { method, headers: { 'Authorization': `Bearer ${fetchserpToken}`, 'Content-Type': 'application/json', }, }; if (body && method !== 'GET') { fetchOptions.body = JSON.stringify(body); } const response = await fetch(url.toString(), fetchOptions); if (!response.ok) { const errorText = await response.text(); throw new McpError( ErrorCode.InternalError, `API request failed: ${response.status} ${response.statusText} - ${errorText}` ); } return await response.json(); }