firecrawl_generate_llmstxt

Instructions

{
  "name": "firecrawl_generate_llmstxt",
  "arguments": {
    "url": "https://example.com",
    "maxUrls": 20,
    "showFullText": true
  }
}

Implementation Reference

• src/index.ts:1366-1418 (handler)

  The handler for the 'firecrawl_generate_llmstxt' tool. Validates input using isGenerateLLMsTextOptions, calls client.generateLLMsText with retry logic, formats the LLMs.txt and optional llms-full.txt content, and returns the result or error.

  case 'firecrawl_generate_llmstxt': {
    if (!isGenerateLLMsTextOptions(args)) {
      throw new Error('Invalid arguments for firecrawl_generate_llmstxt');
    }
  
    try {
      const { url, ...params } = args;
      const generateStartTime = Date.now();
  
      safeLog('info', `Starting LLMs.txt generation for URL: ${url}`);
  
      // Start the generation process
      const response = await withRetry(
        async () =>
          // @ts-expect-error Extended API options including origin
          client.generateLLMsText(url, { ...params, origin: 'mcp-server' }),
        'LLMs.txt generation'
      );
  
      if (!response.success) {
        throw new Error(response.error || 'LLMs.txt generation failed');
      }
  
      // Log performance metrics
      safeLog(
        'info',
        `LLMs.txt generation completed in ${Date.now() - generateStartTime}ms`
      );
  
      // Format the response
      let resultText = '';
  
      if ('data' in response) {
        resultText = `LLMs.txt content:\n\n${response.data.llmstxt}`;
  
        if (args.showFullText && response.data.llmsfulltxt) {
          resultText += `\n\nLLMs-full.txt content:\n\n${response.data.llmsfulltxt}`;
        }
      }
  
      return {
        content: [{ type: 'text', text: trimResponseText(resultText) }],
        isError: false,
      };
    } catch (error) {
      const errorMessage =
        error instanceof Error ? error.message : String(error);
      return {
        content: [{ type: 'text', text: trimResponseText(errorMessage) }],
        isError: true,
      };
    }
  }

• src/index.ts:640-683 (schema)

  The Tool object defining the schema, name, description, and inputSchema for the firecrawl_generate_llmstxt tool.

  const GENERATE_LLMSTXT_TOOL: Tool = {
    name: 'firecrawl_generate_llmstxt',
    description: `
  Generate a standardized llms.txt (and optionally llms-full.txt) file for a given domain. This file defines how large language models should interact with the site.
  
  **Best for:** Creating machine-readable permission guidelines for AI models.
  **Not recommended for:** General content extraction or research.
  **Arguments:**
  - url (string, required): The base URL of the website to analyze.
  - maxUrls (number, optional): Max number of URLs to include (default: 10).
  - showFullText (boolean, optional): Whether to include llms-full.txt contents in the response.
  **Prompt Example:** "Generate an LLMs.txt file for example.com."
  **Usage Example:**
  \`\`\`json
  {
    "name": "firecrawl_generate_llmstxt",
    "arguments": {
      "url": "https://example.com",
      "maxUrls": 20,
      "showFullText": true
    }
  }
  \`\`\`
  **Returns:** LLMs.txt file contents (and optionally llms-full.txt).
  `,
    inputSchema: {
      type: 'object',
      properties: {
        url: {
          type: 'string',
          description: 'The URL to generate LLMs.txt from',
        },
        maxUrls: {
          type: 'number',
          description: 'Maximum number of URLs to process (1-100, default: 10)',
        },
        showFullText: {
          type: 'boolean',
          description: 'Whether to show the full LLMs-full.txt in the response',
        },
      },
      required: ['url'],
    },
  };

• src/index.ts:962-973 (registration)

  Registration of the firecrawl_generate_llmstxt tool in the list of available tools returned by ListToolsRequestSchema handler.

  server.setRequestHandler(ListToolsRequestSchema, async () => ({
    tools: [
      SCRAPE_TOOL,
      MAP_TOOL,
      CRAWL_TOOL,
      CHECK_CRAWL_STATUS_TOOL,
      SEARCH_TOOL,
      EXTRACT_TOOL,
      DEEP_RESEARCH_TOOL,
      GENERATE_LLMSTXT_TOOL,
    ],
  }));

• src/index.ts:839-848 (helper)

  Type guard helper function used to validate the input arguments for the firecrawl_generate_llmstxt tool.

  function isGenerateLLMsTextOptions(
    args: unknown
  ): args is { url: string } & Partial<GenerateLLMsTextParams> {
    return (
      typeof args === 'object' &&
      args !== null &&
      'url' in args &&
      typeof (args as { url: unknown }).url === 'string'
    );
  }