analyze_sitemap
Analyze robots.txt and XML sitemaps to identify blocking issues, discover site structure, check URL counts, and assess sitemap freshness for technical SEO auditing.
Instructions
Analyze a site's robots.txt and XML sitemaps.
Returns:
robots.txt rules and any blocking issues
All discovered sitemaps (from robots.txt and common locations)
URL counts and job-specific URL detection
Sitemap freshness analysis
Recommendations for job boards (Indexing API, job sitemaps)
Use this as a first step to understand site structure before crawling.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| baseUrl | Yes | The base URL of the site (e.g., https://example.com) | |
| includeSitemapUrls | No | Include full URL list from sitemaps (default: true) | |
| maxUrls | No | Maximum URLs to process per sitemap (default: 1000) |