scrape_webpage
Extract raw webpage content without JavaScript execution using the FetchSERP MCP Server, enabling precise web scraping for SEO, data analysis, or keyword research tasks.
Instructions
Scrape a web page without JS
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The url to scrape |
Implementation Reference
- index.js:644-645 (handler)Handler implementation for the 'scrape_webpage' tool. It calls the makeRequest method with the specific API endpoint '/api/v1/scrape' using GET method and the provided arguments.
case 'scrape_webpage': return await this.makeRequest('/api/v1/scrape', 'GET', args, null, token); - index.js:251-262 (schema)Input schema definition for the 'scrape_webpage' tool, including name, description, and required 'url' parameter.
name: 'scrape_webpage', description: 'Scrape a web page without JS', inputSchema: { type: 'object', properties: { url: { type: 'string', description: 'The url to scrape', }, }, required: ['url'], }, - index.js:250-262 (registration)Registration of the 'scrape_webpage' tool in the ListTools response, defining its metadata and schema.
{ name: 'scrape_webpage', description: 'Scrape a web page without JS', inputSchema: { type: 'object', properties: { url: { type: 'string', description: 'The url to scrape', }, }, required: ['url'], }, - index.js:565-613 (helper)Helper method 'makeRequest' used by the scrape_webpage handler to perform authenticated HTTP requests to the FetchSERP API.
async makeRequest(endpoint, method = 'GET', params = {}, body = null, token = null) { const fetchserpToken = token || process.env.FETCHSERP_API_TOKEN; if (!fetchserpToken) { throw new McpError( ErrorCode.InvalidRequest, 'FETCHSERP_API_TOKEN is required' ); } const url = new URL(`${API_BASE_URL}${endpoint}`); // Add query parameters for GET requests if (method === 'GET' && Object.keys(params).length > 0) { Object.entries(params).forEach(([key, value]) => { if (value !== undefined && value !== null) { if (Array.isArray(value)) { value.forEach(v => url.searchParams.append(`${key}[]`, v)); } else { url.searchParams.append(key, value.toString()); } } }); } const fetchOptions = { method, headers: { 'Authorization': `Bearer ${fetchserpToken}`, 'Content-Type': 'application/json', }, }; if (body && method !== 'GET') { fetchOptions.body = JSON.stringify(body); } const response = await fetch(url.toString(), fetchOptions); if (!response.ok) { const errorText = await response.text(); throw new McpError( ErrorCode.InternalError, `API request failed: ${response.status} ${response.statusText} - ${errorText}` ); } return await response.json(); }