seo_audit_robots_and_sitemap
Audits robots.txt and sitemap.xml to detect contradictory crawl rules and HTTP security issues, ensuring correct indexation for search engines and AI bots.
Instructions
Inspects robots.txt rules (allow/disallow per user-agent), sitemap index validity, detects contradictory directives (e.g. disallowed pages mistakenly included in sitemap.xml), and audits HTTP security headers.
USAGE GUIDELINES:
Use to inspect crawl configuration and indexation guardrails for Googlebot, GPTBot, ClaudeBot, and PerplexityBot.
Do NOT use for crawling page body content; use 'seo_audit_sitemap_multipage' or 'seo_crawl_and_extract' instead.
BEHAVIORAL TRANSPARENCY:
Safe, read-only network/file inspection. Fetches robots.txt and sitemap.xml without modifying them.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| target | Yes | Website URL (e.g. "https://example.com") or local codebase folder path. |