fetch_txt
Extracts plain text content from a website by fetching the URL, removing HTML, and allowing customization with headers, character limits, and start indexes. Simplify text-only webpage retrieval for streamlined processing.
Instructions
Fetch a website, return the content as plain text (no HTML)
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| headers | No | Optional headers to include in the request | |
| max_length | No | Maximum number of characters to return (default: 5000) | |
| start_index | No | Start content from this character index (default: 0) | |
| url | Yes | URL of the website to fetch |
Implementation Reference
- src/Fetcher.ts:92-125 (handler)Core handler function for 'fetch_txt' tool. Fetches HTML, uses JSDOM to remove scripts/styles, extracts and normalizes body textContent, applies length limits, returns text content or error.static async txt(requestPayload: RequestPayload) { try { const response = await this._fetch(requestPayload); const html = await response.text(); const dom = new JSDOM(html); const document = dom.window.document; const scripts = document.getElementsByTagName("script"); const styles = document.getElementsByTagName("style"); Array.from(scripts).forEach((script) => script.remove()); Array.from(styles).forEach((style) => style.remove()); const text = document.body.textContent || ""; let normalizedText = text.replace(/\s+/g, " ").trim(); // Apply length limits normalizedText = this.applyLengthLimits( normalizedText, requestPayload.max_length ?? 5000, requestPayload.start_index ?? 0 ); return { content: [{ type: "text", text: normalizedText }], isError: false, }; } catch (error) { return { content: [{ type: "text", text: (error as Error).message }], isError: true, }; } }
- src/index.ts:82-108 (registration)Tool registration in ListTools handler, including name, description, and inputSchema definition.{ name: "fetch_txt", description: "Fetch a website, convert the content to plain text (no HTML)", inputSchema: { type: "object", properties: { url: { type: "string", description: "URL of the website to fetch", }, headers: { type: "object", description: "Optional headers to include in the request", }, max_length: { type: "number", description: `Maximum number of characters to return (default: ${downloadLimit})`, }, start_index: { type: "number", description: "Start content from this character index (default: 0)", }, }, required: ["url"], }, },
- src/index.ts:152-155 (registration)Dispatch logic in CallToolRequestSchema handler that routes 'fetch_txt' calls to Fetcher.txt.if (request.params.name === "fetch_txt") { const fetchResult = await Fetcher.txt(validatedArgs); return fetchResult; }
- src/types.ts:5-10 (schema)Zod schema used to validate input arguments for all fetch tools, including 'fetch_txt' (parsed in index.ts before dispatching).export const RequestPayloadSchema = z.object({ url: z.string().url(), headers: z.record(z.string()).optional(), max_length: z.number().int().min(0).optional().default(downloadLimit), start_index: z.number().int().min(0).optional().default(0), });
- src/Fetcher.ts:15-44 (helper)Shared helper method for performing the actual HTTP fetch with security checks (private IP block) and error handling, used by all tool handlers including txt.private static async _fetch({ url, headers, }: RequestPayload): Promise<Response> { try { if (is_ip_private(url)) { throw new Error( `Fetcher blocked an attempt to fetch a private IP ${url}. This is to prevent a security vulnerability where a local MCP could fetch privileged local IPs and exfiltrate data.`, ); } const response = await fetch(url, { headers: { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36", ...headers, }, }); if (!response.ok) { throw new Error(`HTTP error: ${response.status}`); } return response; } catch (e: unknown) { if (e instanceof Error) { throw new Error(`Failed to fetch ${url}: ${e.message}`); } else { throw new Error(`Failed to fetch ${url}: Unknown error`); } } }