Discover all URLs on a website without scraping content. Use for sitemap audits, finding unindexed pages, and understanding site structure.
MIT
440,686 tools. Updated 2026-08-11 02:50
"Understanding Web Scraping" matching MCP tools:
- Extract focused web content with optimized scraping, limited scrolls, and customizable image extraction for efficient data collection on Prysm MCP Server.MIT
- Extract web content from URLs using Bittensor's decentralized scraping network. Pay per use via blockchain for automated data collection.
- Clear stored scraping history to free up memory and ensure privacy by removing past web scraping records.MIT
- Retrieve French tax information from cached data when web scraping fails, providing reliable access to official tax brackets and calculations for residents.
- Control a web browser to click, type, scroll, take screenshots, and run JavaScript. Chain up to 20 actions per call for login flows, form submission, or scraping interactive pages.MIT
Matching MCP Servers
- AlicenseAqualityCmaintenanceA minimal MCP server that exposes Gemini's web and multimodal understanding (video, audio, document, image) through five dedicated tools.51MIT
- AlicenseAqualityAmaintenanceAn MCP server that enables LLMs to drive a Chromium browser for web scraping, collecting structured data, and exporting to CSV.150MIT
Matching MCP Connectors
Generic URL crawl + HTML extraction — fallback for sites without dedicated MCPs.
Web scraping for AI agents. Extract text and metadata from any URL worldwide. $0.005/page.
- Retrieve live bid, ask, and last price for a trading symbol from a connected broker. Get current market quote without web scraping.MIT
- Fetch a public HTTPS URL and return a condensed prose summary with key points, providing a quick understanding of the web page.MIT
- Get instructions for obtaining an API key and using a web-access gateway that connects to 24 search, scraping, browser, and voice providers at provider-metered prices.MIT
- Route requests through datacenter IPs for high-volume scraping of APIs and public data feeds without aggressive anti-bot protections. Supports geo-targeting and sticky IP sessions.MIT
- List saved scraping flows to find flow IDs and discover available flows before running one.MIT
- Ranks web-access providers by measured benchmarks (quality, latency, cost, error rate) for a given job like search, scrape, or crawl. Returns executable routes and honest caveats to help decide which API to use.MIT
- Search academic papers on Google Scholar by query, author, or publication year to find relevant research publications.MIT
- Preview scraping costs for any URL before committing. Check anti-bot tier (1–4), estimated cost, and confidence level to avoid surprises on protected sites.MIT
- Check real-time train ticket prices for routes in Spain. Uses web scraping to provide actual pricing with pagination support.MIT
- Check remaining scrape credits to confirm API key balance before running large scraping jobs.MIT
- Fetch any public web page and extract its readable content as clean markdown. Respects robots.txt, handles redirects, ideal for research and RAG.MIT
- Extract structured data from web pages by specifying instructions and a JSON schema. Ideal for scraping specific content like product details, articles, or lists.Apache 2.0
- Retrieve URLs, titles, and snippets from web search to selectively scrape or filter results yourself.MIT
- Extract clean readable content from up to 20 URLs: text, title, author, date. Eliminates scraping. Feed web pages into AI agent context windows.MIT