"Web scraping and crawling tools" matching MCP servers:

Enables AI assistants to fetch public web pages as readable text or HTML, with caching, live refresh, and parallel job handling.

Enables AI agents to extract structured content from URLs, search the web via DuckDuckGo, monitor site availability, and convert HTML to markdown without requiring API keys.

MCP server exposing web_search, web_fetch, and cluesift_domain_fetch tools with configurable Ollama API keys and HTTP or stdio transport.

Enables web page scraping via Jina reader API and searching FastMCP documentation using minsearch. Supports fetching markdown content from URLs and querying indexed documentation files.

Provides web connectivity tools for searching the web via DuckDuckGo or SerpAPI, fetching URL content, and extracting readable text from web pages.

Provides lightweight web crawling and scraping capabilities, enabling users to fetch pages as clean Markdown, recursively crawl domains, extract metadata, and perform search-and-crawl operations.

Enables ChatGPT plugins and other MCP clients to fetch live web pages and documents as Markdown and to call HTTP APIs via self-hosted Cloudflare Workers.

Provides eight tools for web search, content extraction, screenshots, PDFs, JavaScript execution, crawling, and Wayback Machine snapshots and archiving.

A lightweight MCP server that exposes Crawl4AI web scraping and crawling capabilities as tools for AI agents, enabling single-page scraping and multi-page crawling with adaptive stopping.

Enables a chat session's owner to reach any host through arbitrary HTTP methods with custom headers and bodies, raw TCP sockets, outbound WebSockets, and browser-grade scraping that solves challenges and discloses explicit failover routing.

. A good description should be concise and describe what the server enables users to do, e.g. * Enables to interact with WordPress sites...' Then a huge block? Actually the user message includes the README content inside tags, then after the readme, there is a large block of text starting with '{

A comprehensive web scraping MCP server with 26 tools for fetching, parsing, extracting, and assisting with web content, returning Markdown-formatted results.

Enables web scraping, crawling, structured data extraction, and browser automation through multiple AI agents including OpenAI's CUA, Anthropic's Claude Computer Use, and Browser Use.

Exposes web search, site mapping, scraping, crawling, schema-validated structured extraction, and stateful Playwright browser interaction as MCP tools, routing every call through the same authenticated REST gateway used by the platform's HTTP clients.

An MCP server enabling AI agents to access the internet via web scraping, crawling, searching, and structured data extraction using Firecrawl.

Provides AI agents and RAG pipelines with production-grade web search, deep research, news, academic, extraction, crawling, and source verification tools. Enables citation-anchored, deduplicated, fresh, and security-hardened evidence gathering from across the web.

A self-hosted MCP server offering unlimited web search, scraping, and content extraction tools with full-content reads, no truncation, and zero per-request cost.

Multi-engine web search and content extraction MCP server with 10 sources, 4-way parallel queries, and content extraction via Jina Reader, no API keys required.

MCP server that parses web pages into clean markdown via Crawl4AI's HTTP API, exposing parse_page and health-check tools with configurable proxy and timeout.

Provides web access tools (fetch, search, sitemap crawl) through String AI's API, automatically handling anti-bot bypass, CAPTCHA, and JavaScript rendering.