Update crawler and AI discovery settings
update_crawler_settingsConfigure robots.txt, sitemap.xml, llms.txt, and ai.txt by enabling or disabling crawler access and setting disallowed paths.
Instructions
Control dynamic robots.txt, sitemap.xml, llms.txt and ai.txt generation.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| llmsEnabled | No | ||
| aiTxtEnabled | No | ||
| disallowPaths | No | ||
| robotsEnabled | No | ||
| aiCustomPolicy | No | ||
| sitemapEnabled | No | ||
| allowAiCrawlers | No | ||
| sitemapMaxPosts | No | ||
| llmsIncludePosts | No | ||
| llmsInstructions | No | ||
| llmsIntroduction | No | ||
| aiTrainingAllowed | No | ||
| allowSearchEngines | No | ||
| llmsIncludeServices | No | ||
| sitemapIncludePosts | No | ||
| aiAttributionRequired | No | ||
| llmsIncludeCategories | No | ||
| sitemapIncludeServices | No | ||
| sitemapIncludeCategories | No | ||
| sitemapIncludeNewsletter | No |