fetch_html
Retrieve website content as HTML from any URL using HTTP requests with optional custom headers for web scraping or data extraction.
Instructions
Fetch a website and return the content as HTML
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL of the website to fetch | |
| headers | No | Optional headers to include in the request |
Implementation Reference
- src/Fetcher.ts:32-43 (handler)Implements the core logic for the 'fetch_html' tool: fetches the given URL using _fetch helper and returns the raw HTML content as text, or error message if failed.static async html(requestPayload: RequestPayload) { try { const response = await this._fetch(requestPayload); const html = await response.text(); return { content: [{ type: "text", text: html }], isError: false }; } catch (error) { return { content: [{ type: "text", text: (error as Error).message }], isError: true, }; } }
- src/index.ts:28-45 (registration)Registers the 'fetch_html' tool in the ListToolsRequestSchema handler, providing name, description, and inputSchema.{ name: "fetch_html", description: "Fetch a website and return the content as HTML", inputSchema: { type: "object", properties: { url: { type: "string", description: "URL of the website to fetch", }, headers: { type: "object", description: "Optional headers to include in the request", }, }, required: ["url"], }, },
- src/types.ts:3-6 (schema)Zod schema used to validate input arguments for the fetch_html tool (and others), parsed in the CallToolRequestSchema handler.export const RequestPayloadSchema = z.object({ url: z.string().url(), headers: z.record(z.string()).optional(), });
- src/Fetcher.ts:6-30 (helper)Shared private helper method that performs the actual HTTP fetch request with custom User-Agent and error handling, called by the html handler.private static async _fetch({ url, headers, }: RequestPayload): Promise<Response> { try { const response = await fetch(url, { headers: { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36", ...headers, }, }); if (!response.ok) { throw new Error(`HTTP error: ${response.status}`); } return response; } catch (e: unknown) { if (e instanceof Error) { throw new Error(`Failed to fetch ${url}: ${e.message}`); } else { throw new Error(`Failed to fetch ${url}: Unknown error`); } } }
- src/index.ts:110-112 (handler)Dispatch logic in CallToolRequestSchema handler that routes 'fetch_html' calls to Fetcher.html after validation.if (request.params.name === "fetch_html") { const fetchResult = await Fetcher.html(validatedArgs); return fetchResult;