read_webpage
Extract text content from any webpage by providing its URL. This tool fetches and processes web content for analysis or data collection.
Instructions
Fetch and extract text content from a webpage
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL of the webpage to read |
Implementation Reference
- src/index.ts:206-254 (handler)The main handler logic for the 'read_webpage' tool. Validates input arguments, fetches the webpage using axios (with optional proxy), parses HTML with cheerio to extract title and cleaned body text, structures the output as WebpageContent, and handles errors.} else if (request.params.name === 'read_webpage') { if (!isValidWebpageArgs(request.params.arguments)) { throw new McpError( ErrorCode.InvalidParams, 'Invalid webpage arguments' ); } const { url } = request.params.arguments; try { const proxyConfig = createProxyConfig(); const response = await axios.get(url, { proxy: proxyConfig, }); const $ = cheerio.load(response.data); // Remove script and style elements $('script, style').remove(); const content: WebpageContent = { title: $('title').text().trim(), text: $('body').text().trim().replace(/\s+/g, ' '), url: url, }; return { content: [ { type: 'text', text: JSON.stringify(content, null, 2), }, ], }; } catch (error) { if (axios.isAxiosError(error)) { return { content: [ { type: 'text', text: `Webpage fetch error: ${error.message}`, }, ], isError: true, }; } throw error; } }
- src/index.ts:140-153 (registration)Registration of the 'read_webpage' tool in the MCP server's ListTools response, defining its name, description, and input schema.{ name: 'read_webpage', description: 'Fetch and extract text content from a webpage', inputSchema: { type: 'object', properties: { url: { type: 'string', description: 'URL of the webpage to read', }, }, required: ['url'], }, },
- src/index.ts:73-78 (schema)Input validation type guard (schema) for the 'read_webpage' tool arguments, ensuring the presence of a valid 'url' string.const isValidWebpageArgs = ( args: any ): args is { url: string } => typeof args === 'object' && args !== null && typeof args.url === 'string';
- src/index.ts:59-62 (schema)TypeScript interface defining the structure of the webpage content output from the 'read_webpage' tool.interface WebpageContent { title: string; text: string; url: string;
- src/index.ts:18-42 (helper)Helper function to create Axios proxy configuration from environment variables, used in the 'read_webpage' handler for HTTP requests.function createProxyConfig(): AxiosProxyConfig | false { const httpsProxy = process.env.HTTPS_PROXY || process.env.https_proxy; const httpProxy = process.env.HTTP_PROXY || process.env.http_proxy; const proxyUrl = httpsProxy || httpProxy; if (!proxyUrl) { return false; } try { const url = new URL(proxyUrl); return { protocol: url.protocol.replace(':', ''), host: url.hostname, port: parseInt(url.port) || (url.protocol === 'https:' ? 443 : 80), auth: url.username && url.password ? { username: url.username, password: url.password } : undefined }; } catch (error) { console.warn(`Invalid proxy URL: ${proxyUrl}`); return false; }