Crawl a website to gather content from multiple pages. Returns a job ID for async polling. Best for whole-site extraction; for single pages use scrape, for URL discovery use map.
MIT
133,382 tools. Last updated 2026-05-25 16:29
"How to scrape content from a website" matching MCP tools:
- Discover all indexed URLs on a website to identify pages for scraping or locate specific content when scrape results are incomplete.
- Discover all indexed URLs on a website to identify pages for scraping or locate specific site sections. Returns a list of found URLs.MIT
- Discover all indexed URLs on a website to identify pages for scraping or locate specific sections before content extraction.MIT
- Extract content from all pages on a website by creating a crawl job. Returns a job ID to check status and retrieve structured data results.MIT
- Perform real-time web searches and scrape content from URLs using AI-powered search. Configure result counts and crawl modes to retrieve relevant website information.MIT
Matching MCP Servers
- FlicenseBqualityCmaintenanceFetches website content and converts it to Markdown format with AI-powered content cleanup, ad removal, and full OpenAPI/Swagger specification support for easy processing by AI assistants.Last updated4253
- AlicenseBqualityDmaintenanceAn AI recipe recommendation server based on the MCP protocol, providing functions such as recipe query, classification filtering, intelligent dietary planning, and daily menu recommendation.Last updated651Apache 2.0
Matching MCP Connectors
Transform any blog post or article URL into ready-to-post social media content for Twitter/X threads, LinkedIn posts, Instagram captions, Facebook posts, and email newsletters. Pay-per-event: $0.07 for all 5 platforms, $0.03 for single platform.
Improve security writing, score it against rubrics, plan IR and product strategy.
- Perform real-time web searches with configurable parameters to retrieve and scrape content from relevant websites for AI assistants.MIT
- Generate a standardized LLMs.txt file for any URL to define how large language models should interact with the website content.MIT
- Create documents in Pickaxe knowledge bases by adding raw text content or scraping website URLs to build AI agent resources.MIT
- Scrape any website and get clean Markdown output. Specify location, locale, and rendering options to extract content tailored to your needs.ISC
- Scrape content from various source types by providing a source type and name; the tool delegates to the appropriate module for extraction.MIT
- Ingest website content from sitemap.xml for knowledge management. Filter by paths and set link limits to organize content in SourceSync.ai.MIT
- Scrape website content from a URL, save it to cloud storage, and automatically create embeddings for AI applications.
- Remove all scraped pages from a specific website, clearing content for a fresh re-scrape.MIT
- Scrape website documentation using automated sub-agents. Supports CSS selectors and URL filtering for targeted content extraction.MIT
- Scrape ChatGPT or Gemini search results for a keyword to analyze AI-generated local SEO insights. Costs 3 credits per query.MIT
- Scrape B2B leads from Apollo.io by submitting a search URL. Returns a runId to check status later. Use webhooks for async delivery instead of polling.MIT
- Scrape commercial real estate listings from Crexi by providing search result URLs. Extract property data for lead generation and market analysis.MIT
- Scrape Google search results with customizable pagination, geolocation, locale, and output formats. Parses content into structured data for efficient extraction.MIT
- Retrieve a list of all blog posts for a given website ID. Use this tool to view existing posts before creating or updating content.MIT