Search the web to find relevant pages, returning URLs, titles, and snippets immediately for pre-scraping discovery.
Apache 2.0
679,131 tools. Updated 2026-10-19 23:37
"Web scraping and crawling tools" matching MCP tools:
- Search Apify and Crawlee docs via full-text queries to find relevant pages on platform features, SDKs, CLI, REST API, and web scraping libraries.MIT
- Discover available integrations and apps in Agentled to identify tools for building workflows, including LinkedIn enrichment, email finder, web scraping, and CRM connections.MIT
- Search Apify and Crawlee documentation by keyword to find relevant platform, SDK, CLI, API, or web scraping guides, then fetch full pages via URL.MIT
- Convert HTML to clean Markdown with GFM support for tables and strikethrough. Ideal for scraping web content for LLM processing.MIT
- Search web pages and retrieve content with optional scraping. Get SERP results or extract full page content in formats like markdown and HTML.MIT
Matching MCP Servers
- FlicenseAqualityAmaintenanceEnables AI assistants to fetch public web pages as readable text or HTML, with caching, live refresh, and parallel job handling.2-
- AlicenseAqualityAmaintenanceEnables fetching any public web page through managed proxies, with optional JavaScript rendering, returning markdown, text, HTML, or structured JSON.137 npm33 PyPI11MIT
Matching MCP Connectors
- web-scrapingOAuth
Web scraping for AI agents: scrape, search, crawl, map any website to markdown + JSON. No browser.
11 tools: web pages, JS apps and PDFs as text, feeds, sitemaps, search. 25 free calls.
- Discover web application endpoints and parameters by crawling with Katana, revealing parameterized URLs for further vulnerability scanning with Dalfox and SQLMap.MIT
- Analyze a website's robots.txt file to determine crawl permissions and ensure compliance with ethical web scraping practices. Provides insights into allowed and disallowed paths for crawling.MIT
- Find freelance web scraping and browser automation leads by scanning tech subreddits for pain points and urgency signals.MIT
- Retrieve French tax information from cached data when web scraping fails, providing reliable access to official tax brackets and calculations for residents.Apache 2.0
- Extract focused web content with optimized scraping, limited scrolls, and customizable image extraction for efficient data collection on Prysm MCP Server.MIT
- Extract and summarize web content from URLs. Read pages, crawl sites, or pull structured data, focusing results on your query.MIT
- Conduct comprehensive web research by crawling, searching, and analyzing content to gather detailed information on any topic.MIT
- Conduct deep web research on any query using crawling, search, and AI analysis to gather comprehensive information from multiple sources.MIT
- Generate critical instructions for web scraping tools to handle proxy rotation and bypass anti-bot measures when extracting data from websites.ISC
- Extract comprehensive web content, including images, using deep scraping techniques with customizable parameters such as scroll depth, image size, and pagination. Output data to a specified directory for thorough analysis.MIT
- Extract web content from URLs using Bittensor's decentralized scraping network. Pay per use via blockchain for automated data collection.-
- Clear stored scraping history to free up memory and ensure privacy by removing past web scraping records.MIT
- Generate stealth web scraping scripts using Playwright/Puppeteer, user-agent spoofing, and headless evasion to deliver structured JSON data.MIT
- Create an analysis suite and launch an AI crawler to map a web app's structure from a starting URL, combining suite creation and crawling in one step.MIT
- Start multi-page web crawling to extract structured data with AI or convert content to markdown from a starting URL.MIT