fetch-url-docs
Fetch documentation from library URLs across multiple programming languages, extracting READMEs, API docs, and code examples for analysis without requiring API keys.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL of the library documentation to fetch |
Implementation Reference
- src/index.ts:138-170 (handler)Handler function for the 'fetch-url-docs' tool. Takes a URL, fetches documentation using scraperService.fetchLibraryDocumentation, and returns the content or error.async ({ url }) => { console.error(`Fetching documentation from URL: ${url}`); try { const documentationContent = await scraperService.fetchLibraryDocumentation(url); return { content: [ { type: "text", text: documentationContent, }, ], }; } catch (error) { console.error("Error fetching URL content:", error); const errorMessage = `Error fetching URL content: ${ error instanceof Error ? error.message : String(error) }`; return { content: [ { type: "text", text: errorMessage, }, ], isError: true, }; } }
- src/index.ts:135-137 (schema)Zod schema defining the input parameter 'url' as a valid URL string.{ url: z.string().url().describe("URL of the library documentation to fetch"), },
- src/index.ts:133-171 (registration)Registration of the 'fetch-url-docs' tool on the MCP server, specifying name, input schema, and handler function.server.tool( "fetch-url-docs", { url: z.string().url().describe("URL of the library documentation to fetch"), }, async ({ url }) => { console.error(`Fetching documentation from URL: ${url}`); try { const documentationContent = await scraperService.fetchLibraryDocumentation(url); return { content: [ { type: "text", text: documentationContent, }, ], }; } catch (error) { console.error("Error fetching URL content:", error); const errorMessage = `Error fetching URL content: ${ error instanceof Error ? error.message : String(error) }`; return { content: [ { type: "text", text: errorMessage, }, ], isError: true, }; } } );
- Key helper method in ScraperService that performs the actual documentation fetching, crawling up to maxPages, processing, compiling into Markdown, and adding prompt instructions.public async fetchLibraryDocumentation( url: string, maxPages = 5 ): Promise<string> { try { // If input is not a URL, assume it's a package name if (!url.startsWith("http")) { url = `https://www.npmjs.com/package/${url}`; } // Extract library name from URL const libraryName = extractLibraryName(url); // Crawl documentation const pages = await this.crawlDocumentation(url, libraryName, maxPages); if (pages.length === 0) { throw new Error(`Failed to fetch documentation from ${url}`); } // Compile documentation into a single markdown document const documentation = this.compileDocumentation(pages, libraryName); // Include instructions for using the prompt const promptInstructions = ` --- š For better summarization, use the "summarize-library-docs" prompt with: - libraryName: "${libraryName}" - documentation: <the content above> Example: @summarize-library-docs with libraryName="${libraryName}" `; return documentation + promptInstructions; } catch (error) { console.error(`Error fetching URL content:`, error); // Extract library name from URL const libraryName = extractLibraryName(url); const errorMessage = `Error fetching URL content: ${ error instanceof Error ? error.message : String(error) }`; // Include error-specific prompt instructions const promptInstructions = ` --- š For information about this library despite the fetch error, use the "summarize-library-docs" prompt with: - libraryName: "${libraryName}" - errorStatus: "${error instanceof Error ? error.message : String(error)}" Example: @summarize-library-docs with libraryName="${libraryName}" and errorStatus="fetch failed" `; return errorMessage + promptInstructions; } }