F
licenseNot graded
qualityA
maintenanceEnables AI agents and LLM pipelines to fetch webpages or crawl sites locally and convert the content into clean, token-efficient Markdown with links preserved. Includes caching, robots.txt-aware rate limiting, concurrent depth crawling, and optional chunking for downstream processing.
-