crawl_stats
Analyze server logs to report daily crawl requests, download size, response time, and breakdowns by response, file type, Googlebot type, purpose, and host, plus robots.txt fetches and example URLs.
Instructions
Crawl stats from your logs: requests, download size, response time per day; breakdowns by response, file type, Googlebot type, purpose and host; robots.txt fetches; example URLs.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| host | No | Host, or omit/'all' for every imported host | |
| end_date | No | YYYY-MM-DD (Pacific Time) | |
| examples | No | Example URLs per response type | |
| response | No | Limit top_urls to one response, e.g. 'Not found (404)' | |
| top_urls | No | Most-crawled URLs | |
| bot_types | No | e.g. ['Smartphone','Desktop','Image','AdsBot','StoreBot'] | |
| start_date | No | YYYY-MM-DD (Pacific Time) | |
| verification | No | Which hits count; fake = Google user agent from a non-Google IP | exclude_fake |