fetch-url-docs

Implementation Reference

• src/index.ts:138-170 (handler)

  Handler function for the 'fetch-url-docs' tool. Takes a URL, fetches documentation using scraperService.fetchLibraryDocumentation, and returns the content or error.

  async ({ url }) => {
    console.error(`Fetching documentation from URL: ${url}`);
  
    try {
      const documentationContent =
        await scraperService.fetchLibraryDocumentation(url);
  
      return {
        content: [
          {
            type: "text",
            text: documentationContent,
          },
        ],
      };
    } catch (error) {
      console.error("Error fetching URL content:", error);
  
      const errorMessage = `Error fetching URL content: ${
        error instanceof Error ? error.message : String(error)
      }`;
  
      return {
        content: [
          {
            type: "text",
            text: errorMessage,
          },
        ],
        isError: true,
      };
    }
  }

• src/index.ts:135-137 (schema)

  Zod schema defining the input parameter 'url' as a valid URL string.

  {
    url: z.string().url().describe("URL of the library documentation to fetch"),
  },

• src/index.ts:133-171 (registration)

  Registration of the 'fetch-url-docs' tool on the MCP server, specifying name, input schema, and handler function.

  server.tool(
    "fetch-url-docs",
    {
      url: z.string().url().describe("URL of the library documentation to fetch"),
    },
    async ({ url }) => {
      console.error(`Fetching documentation from URL: ${url}`);
  
      try {
        const documentationContent =
          await scraperService.fetchLibraryDocumentation(url);
  
        return {
          content: [
            {
              type: "text",
              text: documentationContent,
            },
          ],
        };
      } catch (error) {
        console.error("Error fetching URL content:", error);
  
        const errorMessage = `Error fetching URL content: ${
          error instanceof Error ? error.message : String(error)
        }`;
  
        return {
          content: [
            {
              type: "text",
              text: errorMessage,
            },
          ],
          isError: true,
        };
      }
    }
  );

• src/services/ScraperService.ts:137-195 (helper)

  Key helper method in ScraperService that performs the actual documentation fetching, crawling up to maxPages, processing, compiling into Markdown, and adding prompt instructions.

    public async fetchLibraryDocumentation(
      url: string,
      maxPages = 5
    ): Promise<string> {
      try {
        // If input is not a URL, assume it's a package name
        if (!url.startsWith("http")) {
          url = `https://www.npmjs.com/package/${url}`;
        }
  
        // Extract library name from URL
        const libraryName = extractLibraryName(url);
  
        // Crawl documentation
        const pages = await this.crawlDocumentation(url, libraryName, maxPages);
  
        if (pages.length === 0) {
          throw new Error(`Failed to fetch documentation from ${url}`);
        }
  
        // Compile documentation into a single markdown document
        const documentation = this.compileDocumentation(pages, libraryName);
  
        // Include instructions for using the prompt
        const promptInstructions = `
  ---
  
  🔍 For better summarization, use the "summarize-library-docs" prompt with:
  - libraryName: "${libraryName}"
  - documentation: <the content above>
  
  Example: @summarize-library-docs with libraryName="${libraryName}"
        `;
  
        return documentation + promptInstructions;
      } catch (error) {
        console.error(`Error fetching URL content:`, error);
  
        // Extract library name from URL
        const libraryName = extractLibraryName(url);
  
        const errorMessage = `Error fetching URL content: ${
          error instanceof Error ? error.message : String(error)
        }`;
  
        // Include error-specific prompt instructions
        const promptInstructions = `
  ---
  
  🔍 For information about this library despite the fetch error, use the "summarize-library-docs" prompt with:
  - libraryName: "${libraryName}"
  - errorStatus: "${error instanceof Error ? error.message : String(error)}"
  
  Example: @summarize-library-docs with libraryName="${libraryName}" and errorStatus="fetch failed"
        `;
  
        return errorMessage + promptInstructions;
      }
    }