Enables web crawling and content extraction from web pages, supporting multiple output formats like text, markdown, XML, and JSON, with robots.txt compliance and rate limiting.
Enables web scraping of URLs, including extracting text, links, images, metadata, taking screenshots, and using CSS selectors for structured data extraction.
Enables querying and retrieving content from webpages by parsing sitemap.xml files and fetching HTML content from specified URLs. Includes rate limiting for abuse protection.