Enables AI agents to check whether a public website is crawlable, understandable, and ready for AI search workflows through local-only audits of robots.txt, sitemaps, metadata, and llms.txt.
Enables inspection of any website's AI-search readiness, checking AI crawler blocks, llms.txt, schema markup, and indexing directives from MCP clients like Claude.
Audits a domain to determine if AI agents and crawlers can access and read it, returning a flat row of 42 fields covering llms.txt, robots.txt AI policy, structured data, render mode, and machine-readable endpoint discovery.
Enables web content scanning and analysis by fetching, analyzing, and extracting information from web pages using tools like page fetching, link extraction, site crawling, and more.
Analyze and generate robots.txt files with AI crawler awareness. Fetch any site's robots.txt, detect which AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) are blocked or allowed, and generate optimized robots.txt with toggle controls for 20+ AI crawlers.
