Analyze and generate robots.txt files with AI crawler awareness. Fetch any site's robots.txt, detect which AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) are blocked or allowed, and generate optimized robots.txt with toggle controls for 20+ AI crawlers.
Enables natural language browser automation through Claude, wrapping Playwright to execute commands like navigation, clicking, form filling, and screenshots.
Enables fetching daily AI news summaries aggregated from multiple sources including GitHub, Hacker News, Product Hunt, YouTube, Twitter, and Chinese AI sites.
An MCP server for lightweight browser automation using Lightpanda with Chrome fallback, providing LLM-optimized accessibility tree snapshots for precise element targeting.
Wraps the Kimi Coding Search and Fetch APIs into MCP tools for web searching and content retrieval. It enables LLMs to perform targeted searches and crawl web pages using standardized interfaces.
Enables web content extraction, screenshot capture, web search, arXiv paper search, and image search through Jina AI's APIs. Provides tools for reading URLs as markdown, searching the web for current information, and finding academic papers or images.
Enables AI agents to check whether a public website is crawlable, understandable, and ready for AI search workflows through local-only audits of robots.txt, sitemaps, metadata, and llms.txt.