fetch_link_documentation
Extract documentation components, APIs, and usage examples from website links using configurable crawling depth and content selectors.
Instructions
Fetch and analyze documentation from a website link, extracting all available components, APIs, and usage examples
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| depth | No | How deep to crawl links (0-3, default: 1) | |
| selector | No | Optional CSS selector to focus on specific content | body |
| url | Yes | The URL to fetch documentation from (e.g., WindUI documentation) |
Implementation Reference
- src/tools/linkDocumentationTool.ts:24-41 (handler)The entry-point handler function for the 'fetch_link_documentation' tool. It clears visited URLs, crawls the provided URL (with optional selector and depth), formats the result, and returns it as MCP content.async fetchDocumentation(args: FetchDocumentationArgs) { this.visitedUrls.clear(); try { const result = await this.crawlPage(args.url, args.selector || 'body', args.depth || 1); return { content: [ { type: "text", text: this.formatDocumentationResult(result), }, ], }; } catch (error) { throw new Error(`Failed to fetch documentation: ${error instanceof Error ? error.message : String(error)}`); } }
- Core helper method that performs the actual web crawling, content extraction (title, text, components, APIs, examples, links), and recursive crawling based on depth.private async crawlPage(url: string, selector: string, depth: number): Promise<DocumentationResult> { if (this.visitedUrls.has(url) || depth < 0) { return this.createEmptyResult(url); } this.visitedUrls.add(url); try { const response = await axios.get(url, { timeout: 10000, headers: { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36' } }); const $ = cheerio.load(response.data); const selectedContent = $(selector); const result: DocumentationResult = { url, title: $('title').text().trim() || $('h1').first().text().trim() || 'Untitled', content: this.extractTextContent(selectedContent), components: this.extractComponents($), apis: this.extractAPIs($), examples: this.extractExamples($), links: this.extractLinks($, url), }; // 如果depth > 0，继续爬取相关链接 if (depth > 0) { const relatedLinks = result.links .filter(link => this.isRelevantLink(link)) .slice(0, 5); // 限制每页最多5个链接 for (const link of relatedLinks) { const subResult = await this.crawlPage(link, selector, depth - 1); result.content += `\n\n--- Linked Page: ${subResult.title} ---\n${subResult.content}`; result.components.push(...subResult.components); result.apis.push(...subResult.apis); result.examples.push(...subResult.examples); } } return result; } catch (error) { console.error(`Error crawling ${url}:`, error); return this.createEmptyResult(url); } }
- src/index.ts:39-58 (schema)JSON schema defining the input parameters for the tool: required 'url', optional 'selector' and 'depth'.inputSchema: { type: "object", properties: { url: { type: "string", description: "The URL to fetch documentation from (e.g., WindUI documentation)", }, selector: { type: "string", description: "Optional CSS selector to focus on specific content", default: "body" }, depth: { type: "number", description: "How deep to crawl links (0-3, default: 1)", default: 1 } }, required: ["url"], },
- src/index.ts:33-62 (registration)Registers the tool in MCP ListToolsRequestHandler, providing name, description, and input schema.this.server.setRequestHandler(ListToolsRequestSchema, async () => { return { tools: [ { name: "fetch_link_documentation", description: "Fetch and analyze documentation from a website link, extracting all available components, APIs, and usage examples", inputSchema: { type: "object", properties: { url: { type: "string", description: "The URL to fetch documentation from (e.g., WindUI documentation)", }, selector: { type: "string", description: "Optional CSS selector to focus on specific content", default: "body" }, depth: { type: "number", description: "How deep to crawl links (0-3, default: 1)", default: 1 } }, required: ["url"], }, }, ], }; });
- src/index.ts:68-71 (registration)Dispatches calls to 'fetch_link_documentation' to the LinkDocumentationTool instance in the CallToolRequestHandler.switch (name) { case "fetch_link_documentation": return await this.linkTool.fetchDocumentation(args as any);