Provides lightweight web crawling and scraping capabilities, enabling users to fetch pages as clean Markdown, recursively crawl domains, extract metadata, and perform search-and-crawl operations.
Fetches webpages or crawls sites locally, strips noise, and converts content to clean Markdown while preserving links, enabling fast, token-efficient web scraping for AI agents and LLM pipelines.
Fetches webpages and crawls sites locally, stripping noise and converting content into clean, link-preserving Markdown with a minimal token footprint. Supports configurable crawl depth, concurrency, robots.txt compliance, and caching for efficient AI agent and LLM pipelines.
Crawl any website into clean Markdown, search through pages, read full content, and extract structured data using OpenAI, Claude, Gemini, or Grok — with auto-citation and resume support.
Crawl entire websites, sitemaps, and documentation portals into clean, LLM-ready Markdown without ads, cookie popups, or noise for RAG pipelines and AI agents.