"Web scraping and crawling tools" matching MCP servers:

Enables AI assistants to fetch public web pages as readable text or HTML, with caching, live refresh, and parallel job handling.

Enables fetching any public web page through managed proxies, with optional JavaScript rendering, returning markdown, text, HTML, or structured JSON.

Provides web search and content extraction tools for AI assistants, with zero configuration and swappable backends.

Provides web access capabilities for LLMs including search, fetching, content extraction, PDF reading, image viewing, and screenshots.

Provides an LLM client with self-hosted web scraping capabilities including scraping, crawling, mapping, searching, structured data extraction, browser interaction, and deep research, without reliance on paid proxy or CAPTCHA services.

Enables AI agents to extract structured content from URLs, search the web via DuckDuckGo, monitor site availability, and convert HTML to markdown without requiring API keys.

Provides local web search and content fetching capabilities for AI assistants, enabling them to search DuckDuckGo and extract clean text from web pages. All requests originate from the user's machine to ensure direct network control and bypass external proxies.

Provides browser automation and web scraping as MCP tools, enabling autonomous URL ingestion, crawling, extraction, and anti-bot handling with interactive browser control.

Provides web connectivity tools for searching the web via DuckDuckGo or SerpAPI, fetching URL content, and extracting readable text from web pages.

Enables converting any URL or website into clean, LLM-ready Markdown, supporting single-page scraping and full-site crawling for AI agents.

Provides lightweight web crawling and scraping capabilities, enabling users to fetch pages as clean Markdown, recursively crawl domains, extract metadata, and perform search-and-crawl operations.

Efficient web scraping MCP server with a two-step workflow for token-efficient content extraction. Supports previewing HTML structure, targeted scraping, and entire documentation site crawling.

Enables ChatGPT plugins and other MCP clients to fetch live web pages and documents as Markdown and to call HTTP APIs via self-hosted Cloudflare Workers.

Provides eight tools for web search, content extraction, screenshots, PDFs, JavaScript execution, crawling, and Wayback Machine snapshots and archiving.

A custom MCP server that provides AI agents with tools to fetch web pages, extract readable article text, extract structured data by CSS selector, and check robots.txt permissions.

. A good description should be concise and describe what the server enables users to do, e.g. * Enables to interact with WordPress sites...' Then a huge block? Actually the user message includes the README content inside tags, then after the readme, there is a large block of text starting with '{

A comprehensive web scraping MCP server with 26 tools for fetching, parsing, extracting, and assisting with web content, returning Markdown-formatted results.

Exposes web search, site mapping, scraping, crawling, schema-validated structured extraction, and stateful Playwright browser interaction as MCP tools, routing every call through the same authenticated REST gateway used by the platform's HTTP clients.

A Model Context Protocol server that allows LLMs to interact with web content through standardized tools, currently supporting web scraping functionality.

Enables web searching and fetching pages converted to Markdown through local stdio MCP tools, so Claude Code and other MCP clients connected via third-party relays retain internet access instead of failing with a server-side tool protocol error. Includes paginated reading of long articles, SSRF/private-network blocking, and charset handling for legacy Chinese sites.