A
licenseA
qualityB
maintenanceEnables crawling sites from start URLs and retrieving internal and external links, file download links, HTML tables as rows, and lists, with controls for crawl depth, page limits, sitemaps, robots.txt, and proxy mode.
1
MIT