Discover all URLs on a website without scraping content. Use for sitemap audits, finding unindexed pages, and understanding site structure.
MIT
649,985 tools. Updated 2026-10-11 02:29
"Understanding Web Scraping" matching MCP tools:
- Convert HTML to clean Markdown with GFM support for tables and strikethrough. Ideal for scraping web content for LLM processing.MIT
- Find freelance web scraping and browser automation leads by scanning tech subreddits for pain points and urgency signals.MIT
- Retrieve French tax information from cached data when web scraping fails, providing reliable access to official tax brackets and calculations for residents.Apache 2.0
- Extract focused web content with optimized scraping, limited scrolls, and customizable image extraction for efficient data collection on Prysm MCP Server.MIT
- Extract comprehensive web content, including images, using deep scraping techniques with customizable parameters such as scroll depth, image size, and pagination. Output data to a specified directory for thorough analysis.MIT
Matching MCP Servers
- FlicenseAqualityAmaintenanceEnables AI assistants to fetch public web pages as readable text or HTML, with caching, live refresh, and parallel job handling.2-
- AlicenseAqualityAmaintenanceEnables fetching any public web page through managed proxies, with optional JavaScript rendering, returning markdown, text, HTML, or structured JSON.1192 npm11MIT
Matching MCP Connectors
- web-scrapingOAuth
Web scraping for AI agents: scrape, search, crawl, map any website to markdown + JSON. No browser.
40+ web scraping tools from Firecrawl, Bright Data, Jina, Olostep, ScrapeGraph, Notte, and Riveter. Scrape, crawl, screenshot, and extract from any website. Starts at $0.01/call. Get your API key at app.xpay.sh or xpay.tools
- Extract web content from URLs using Bittensor's decentralized scraping network. Pay per use via blockchain for automated data collection.-
- Clear stored scraping history to free up memory and ensure privacy by removing past web scraping records.MIT
- Generate stealth web scraping scripts using Playwright/Puppeteer, user-agent spoofing, and headless evasion to deliver structured JSON data.MIT
- Control a web browser to click, type, scroll, take screenshots, and run JavaScript. Chain up to 20 actions per call for login flows, form submission, or scraping interactive pages.MIT
- Fetch a public HTTPS URL and return a condensed prose summary with key points, providing a quick understanding of the web page.MIT
- Get instructions for obtaining an API key and using a web-access gateway that connects to 24 search, scraping, browser, and voice providers at provider-metered prices.MIT
- List saved scraping flows to find flow IDs and discover available flows before running one.MIT
- Search the web to find relevant pages, returning URLs, titles, and snippets immediately for pre-scraping discovery.Apache 2.0
- Search Apify and Crawlee docs via full-text queries to find relevant pages on platform features, SDKs, CLI, REST API, and web scraping libraries.MIT
- Ranks web-access providers by measured benchmarks (quality, latency, cost, error rate) for a given job like search, scrape, or crawl. Returns executable routes and honest caveats to help decide which API to use.MIT
- List scrape strategies with protection levels and credit estimates to prepare for scraping unknown protected websites.-
- Crawl a website to discover and scrape multiple pages, following links up to a set depth. Filter URLs with regex and use saved or custom scraping configs.MIT
- Retrieve all mapped scraping pages in your Gluedly workspace to review configured scrapes.MIT
- Add a product to your shop catalog using your own data, with no scraping needed. Specify details like title, price, and affiliate URL to create the entry directly.MIT
- Extract clean content from a public URL, returning Markdown, text, metadata, links, JSON-LD schema, and sections for scraping, RAG ingestion, or content analysis.MIT