fetch-url
Retrieve web content from a specified URL using customizable HTTP methods, headers, and response types. Supports handling redirects, timeouts, and parsing text, JSON, or binary data.
Instructions
Fetch content from a URL
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| body | No | Request body for POST/PUT/PATCH requests | |
| followRedirects | No | Whether to follow redirects | |
| headers | No | HTTP headers | |
| method | No | HTTP method | GET |
| responseType | No | How to parse the response | text |
| timeout | No | Request timeout in milliseconds | |
| url | Yes | URL to fetch |
Implementation Reference
- src/index.ts:179-286 (handler)Main handler function that executes the fetch-url tool: handles HTTP requests with undici, supports various methods/headers/body, processes responses into text/json/binary/html-fragment formats, extracts HTML fragments using JSDOM if selector provided.async function handleFetchUrl(args: FetchUrlArgs): Promise<z.infer<typeof CallToolResultSchema>> { try { const { url, method, headers, body, timeout, responseType, followRedirects } = args; // Log the fetch request console.error(`Fetching ${url} with method ${method}`); // Create request options const options: any = { method, headers: headers || {}, body: body || undefined, redirect: followRedirects ? "follow" : "manual", }; if (timeout) { // @ts-ignore - undici specific options options.bodyTimeout = timeout; // @ts-ignore - undici specific options options.headersTimeout = timeout; } // Perform the request const response = await fetch(url, options); // Create a result object with metadata const result: Record<string, any> = { status: response.status, statusText: response.statusText, headers: Object.fromEntries(response.headers.entries()), url: response.url, }; // Process the response based on the requested type switch (responseType) { case "json": try { const jsonContent = await response.json(); result.content = JSON.stringify(jsonContent, null, 2); } catch (e) { throw new Error(`Failed to parse response as JSON: ${(e as Error).message}`); } break; case "binary": const buffer = await response.arrayBuffer(); result.content = Buffer.from(buffer).toString("base64"); result.encoding = "base64"; break; case "html-fragment": try { const htmlContent = await response.text(); // Use JSDOM to parse the HTML const dom = new JSDOM(htmlContent); const document = dom.window.document; // Only look for fragments if a selector is provided if (args.fragmentSelector) { const elements = document.querySelectorAll(args.fragmentSelector); if (elements.length === 0) { throw new Error(`No elements found matching selector "${args.fragmentSelector}"`); } // Extract the HTML from the selected element(s) if (elements.length === 1) { result.content = elements[0].outerHTML; } else { result.content = Array.from(elements).map(el => el.outerHTML).join('\n'); } result.matchCount = elements.length; } else { // No selector provided, return the full HTML result.content = htmlContent; } result.contentType = 'text/html'; } catch (e) { throw new Error(`Failed to parse or extract HTML fragment: ${(e as Error).message}`); } break; case "text": default: result.content = await response.text(); break; } return { content: [ { type: "text", text: JSON.stringify(result, null, 2) } ] }; } catch (error) { console.error(`Error fetching URL:`, error); return { isError: true, content: [ { type: "text", text: `Error fetching URL: ${(error as Error).message}` } ] }; } }
- src/index.ts:16-25 (schema)Zod schema used to validate and parse arguments for the fetch-url tool before calling the handler.const FetchUrlArgsSchema = z.object({ url: z.string().url().describe("URL to fetch"), method: z.enum(["GET", "POST", "PUT", "DELETE", "HEAD", "OPTIONS", "PATCH"]).default("GET").describe("HTTP method"), headers: z.record(z.string()).optional().describe("HTTP headers"), body: z.string().optional().describe("Request body for POST/PUT/PATCH requests"), timeout: z.number().positive().optional().describe("Request timeout in milliseconds"), responseType: z.enum(["text", "json", "binary", "html-fragment"]).default("text").describe("How to parse the response"), followRedirects: z.boolean().default(true).describe("Whether to follow redirects"), fragmentSelector: z.string().optional().describe("CSS selector for the HTML fragment to extract (when responseType is html-fragment)") });
- src/index.ts:85-119 (registration)Tool registration in the TOOLS array: defines name, description, and inputSchema for listTools handler.{ name: "fetch-url", description: "Fetch content from a URL", inputSchema: { type: "object", properties: { url: { type: "string", description: "URL to fetch" }, method: { type: "string", enum: ["GET", "POST", "PUT", "DELETE", "HEAD", "OPTIONS", "PATCH"], default: "GET", description: "HTTP method" }, headers: { type: "object", additionalProperties: { type: "string" }, description: "HTTP headers" }, body: { type: "string", description: "Request body for POST/PUT/PATCH requests" }, timeout: { type: "number", description: "Request timeout in milliseconds" }, responseType: { type: "string", enum: ["text", "json", "binary", "html-fragment"], default: "text", description: "How to parse the response" }, followRedirects: { type: "boolean", default: true, description: "Whether to follow redirects" } }, required: ["url"] } },
- src/index.ts:159-161 (registration)Dispatch registration in the CallToolRequestSchema handler: parses args with schema and calls the fetch-url handler function.case "fetch-url": return handleFetchUrl(FetchUrlArgsSchema.parse(args)); case "extract-html-fragment":