"Web scraping and crawling tools" matching MCP servers:

Enables AI assistants to fetch public web pages as readable text or HTML, with caching, live refresh, and parallel job handling.

Provides web search and content extraction tools for AI assistants, with zero configuration and swappable backends.

Enables AI agents to extract structured content from URLs, search the web via DuckDuckGo, monitor site availability, and convert HTML to markdown without requiring API keys.

Provides lightweight web crawling and scraping capabilities, enabling users to fetch pages as clean Markdown, recursively crawl domains, extract metadata, and perform search-and-crawl operations.

Provides eight tools for web search, content extraction, screenshots, PDFs, JavaScript execution, crawling, and Wayback Machine snapshots and archiving.

Provides web access tools (fetch, search, sitemap crawl) through String AI's API, automatically handling anti-bot bypass, CAPTCHA, and JavaScript rendering.

Drop-in replacement for Claude's native web_search and web_fetch tools, resolving IP blocks and user-agent detection by using your own API keys and proxy.

Bridges the editing capabilities of an already-open DingTalk document in the browser to any MCP host via standard stdio, forwarding tools to the page's internal MCP server over a local WebSocket. Supports read/write operations through suggested edits, pairing-based security, and zero runtime dependencies.

A Model Context Protocol server that provides real-time web search capabilities to AI assistants through pluggable search providers, currently integrated with the Brave Search API.

A fast, dependency-light MCP server that converts web pages into clean Markdown, structured metadata, and classified links for language models.

Enables web searches through DuckDuckGo without an API key and extracts readable main content from web pages.

An MCP server for web search, web fetching, and Context7 documentation tools with proxy support, designed for internal network environments.

An MCP server that runs Apify scraping actors with quality-gated validation, scoring, and retries, returning embedding-ready chunks for AI agents.

Enables agents to search the live web and fetch URLs as clean Markdown, returning ranked search results and paginated page content through two lightweight tools.

MCP server providing browser compatibility data for the entire Web Platform.

Wraps Xiaomi MiMo's web_search API into an MCP tool for Claude Code, enabling real-time web searches with location awareness, automatic retry, and timeout control.

An APAC-native web scraping API for AI agents that provides tools for scraping, crawling, searching, and extracting structured data from websites, directly usable from MCP-compatible clients like Claude Desktop, Cursor, and Windsurf.

Provides local LLMs with web search and page fetching capabilities via MCP, with a focus on OWASP security best practices.

TypeScript implementation of an MCP server that provides tools for web crawling, content extraction, and browser automation, enabling AI systems to access and process web content through 15 specialized tools.

An MCP server that exposes 43 domain tools behind 4 meta-tools, applying guardrails such as budgets, rate limits, schema validation, human-in-the-loop confirmation for mutations, PII redaction, and an append-only audit log while reducing context window usage.