robots.txt generator
robots_txt_generatorBuild a robots.txt: allow or block all, custom disallow paths, AI-crawler blocking, crawl-delay and sitemap. Runs on smart-tools.xyz.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| access | No | allow all, block all, or custom (default allow) | |
| locale | No | Language for the source_url link (default en) | |
| sitemap | No | Sitemap URL (optional) | |
| block_ai | No | Also block common AI crawlers (default false) | |
| disallow | No | Paths to disallow (used when access=custom) | |
| crawl_delay | No | Crawl-delay in seconds (optional) |