Generate llms.txt
generate_llms_txtGenerate llms.txt for any website by crawling its sitemap, grouping pages, and extracting titles and descriptions. Returns formatted content and robots.txt snippet for AI readiness.
Instructions
Generate a properly formatted llms.txt file for a website. Crawls the site's sitemap, groups pages by section, pulls page titles and descriptions, and produces both the llms.txt content and a companion robots.txt snippet. The AI client can then write the returned content to disk in the user's project. Requires an API key.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The website to generate llms.txt for. |