fetch_llms_txt

Instructions

Implementation Reference

• src/tools/fetch_llms_txt.ts:25-80 (handler)

  The primary handler function for the 'fetch_llms_txt' tool. It normalizes input URLs, validates and parses targets, checks domain access, fetches content from each URL, and returns an array of TextContent results.

  export const fetch_llms_txt = async (
    input: UrlFetchInput,
    extra: RequestHandlerExtra<ServerRequest, ServerNotification>,
    allowedDomains: Set<string>
  ): Promise<CallToolResult> => {
    logger.debug("Processing fetch_llms_txt request with input:", input);
  
    // Normalize input to always be an array of { url: string } objects
    const urlList = normalizeUrlInput(input);
    const results: TextContent[] = [];
  
    for (const urlItem of urlList) {
      const url = urlItem.url;
  
      try {
        logger.debug(`Processing URL: ${url}`);
  
        // Validate the URL and get target info using the library function
        const targetInfo = await parseFetchTarget(url);
        if (targetInfo.type === "unsupported") {
          const errorMsg = `Unsupported URL format: ${targetInfo.reason}`;
          logger.error(errorMsg);
          throw new Error(errorMsg);
        }
  
        logger.debug(`Target info:`, targetInfo);
  
        // Check domain access using the library function
        checkDomainAccess(targetInfo, allowedDomains);
  
        logger.debug(`Fetching content from: ${url}`);
  
        // Fetch the content using the library function
        const fileContent = await fetchContent(targetInfo);
        results.push({
          type: "text",
          text: fileContent,
        });
  
        logger.debug(
          `Successfully fetched ${fileContent.length} bytes from ${url}`
        );
      } catch (error) {
        const errorMsg = `Failed to process fetch request for ${url}: ${
          error instanceof Error ? error.message : String(error)
        }`;
        logger.error(errorMsg);
        throw new Error(errorMsg);
      }
    }
  
    logger.debug(`Successfully processed ${results.length} URLs`);
    return {
      content: results,
    };
  };

• src/tools/tool_schemas.ts:1-32 (schema)

  Zod schema (UrlFetchInputSchema) and type (UrlFetchInput) defining the input validation for the fetch_llms_txt tool, supporting single URL strings, objects, or arrays thereof.

  import { z } from "zod";
  
  /**
   * Schema for URL-based fetch operations that can accept:
   * - A single URL string
   * - A single URL object
   * - An array of URL strings and/or objects
   */
  export const UrlFetchInputSchema = z.union([
    z.string().url("Input must be a valid URL string"),
    z.object({
      url: z
        .string()
        .url("Input must contain a valid URL string under the 'url' key"),
    }),
    z.array(
      z.union([
        z.string().url("Each array item must be a valid URL string"),
        z.object({
          url: z
            .string()
            .url(
              "Each array item must contain a valid URL string under the 'url' key"
            ),
        }),
      ])
    ),
  ]);
  
  // Type exports for use in function parameters
  export type UrlFetchInput = z.infer<typeof UrlFetchInputSchema>;

• src/index.ts:122-140 (registration)

  Registration of the 'fetch_llms_txt' tool using server.tool(), including prompt, input schema reference, and wrapper handler that delegates to the main fetch_llms_txt function.

    "fetch_llms_txt",
    "Fetches the content of one or more llms.txt URLs. Some llms.txt files compile a list of urls to other llms.txt file locations because listing their full documentation would bloat context. If the documentation you're looking for does not exist in the llms.txt, look for reference links to other llms.txt files and follow those.",
    {
      input: UrlFetchInputSchema.describe(
        "URL string, URL object, or array of URL/objects to fetch llms.txt from"
      ),
    },
    async (params, extra) => {
      const input = params?.input ?? params;
      if (!input) {
        throw new Error("No input provided to fetch_llms_txt");
      }
      return fetch_llms_txt(
        input,
        extra as RequestHandlerExtra<ServerRequest, ServerNotification>,
        allowedDomains
      );
    }
  );

• src/index.ts:78-89 (registration)

  Tool capability declaration in the MCP server constructor, specifying name, description, input schema, and annotations for 'fetch_llms_txt'.

  fetch_llms_txt: {
    name: "fetch_llms_txt",
    description: "Fetches the content of a llms.txt url.",
    inputSchema: UrlFetchInputSchema,
    annotations: {
      title: "Fetch llms.txt content",
      readOnlyHint: true,
      destructiveHint: false,
      idempotentHint: true,
      openWorldHint: true,
    },
  },