fetch_markdown
Convert website content into Markdown by fetching the URL. Ideal for extracting and reformatting web data for documentation or analysis.
Instructions
Fetch a website and return the content as Markdown
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| headers | No | Optional headers to include in the request | |
| url | Yes | URL of the website to fetch |
Implementation Reference
- src/Fetcher.ts:90-103 (handler)The core handler function that fetches the HTML content from the provided URL and converts it to Markdown using TurndownService. Handles errors by returning an error message.
static async markdown(requestPayload: RequestPayload) { try { const response = await this._fetch(requestPayload); const html = await response.text(); const turndownService = new TurndownService(); const markdown = turndownService.turndown(html); return { content: [{ type: "text", text: markdown }], isError: false }; } catch (error) { return { content: [{ type: "text", text: (error as Error).message }], isError: true, }; } } - src/index.ts:68-85 (registration)Tool registration in the ListTools response, defining name, description, and input schema.
{ name: "fetch_markdown", description: "Fetch a website and return the content as Markdown", inputSchema: { type: "object", properties: { url: { type: "string", description: "URL of the website to fetch", }, headers: { type: "object", description: "Optional headers to include in the request", }, }, required: ["url"], }, }, - src/types.ts:3-6 (schema)Zod schema used to parse and validate the tool input arguments (url required as valid URL, optional headers).
export const RequestPayloadSchema = z.object({ url: z.string().url(), headers: z.record(z.string()).optional(), }); - src/index.ts:144-147 (handler)Dispatch logic in the CallToolRequest handler that invokes Fetcher.markdown for the fetch_markdown tool.
if (request.params.name === "fetch_markdown") { const fetchResult = await Fetcher.markdown(validatedArgs); return fetchResult; } - src/Fetcher.ts:6-30 (helper)Private helper method used by all fetch tools to perform the actual HTTP fetch with custom User-Agent and error handling.
private static async _fetch({ url, headers, }: RequestPayload): Promise<Response> { try { const response = await fetch(url, { headers: { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36", ...headers, }, }); if (!response.ok) { throw new Error(`HTTP error: ${response.status}`); } return response; } catch (e: unknown) { if (e instanceof Error) { throw new Error(`Failed to fetch ${url}: ${e.message}`); } else { throw new Error(`Failed to fetch ${url}: Unknown error`); } } }