"Web scraping and crawling tools" matching MCP servers:

Provides web access capabilities for LLMs including search, fetching, content extraction, PDF reading, image viewing, and screenshots.

Enables AI agents to extract structured content from URLs, search the web via DuckDuckGo, monitor site availability, and convert HTML to markdown without requiring API keys.

Web search, content extraction, and library docs for AI agents with 5-strategy scraping and runs without API keys.

Comprehensive web research toolkit with 13 tools for searching (via SearXNG), crawling, package discovery, GitHub metrics, error translation, API documentation lookup, data extraction, technology comparison, and service status checking.

Provides local web search and content fetching capabilities for AI assistants, enabling them to search DuckDuckGo and extract clean text from web pages. All requests originate from the user's machine to ensure direct network control and bypass external proxies.

MCP server exposing web_search, web_fetch, and cluesift_domain_fetch tools with configurable Ollama API keys and HTTP or stdio transport.

A powerful tool for fetching and extracting text content from web pages and APIs, supporting web scraping, REST API requests, and Google Custom Search integration.

Built as a Model Context Protocol (MCP) server that provides advanced web search, content extraction, web crawling, and scraping capabilities using the Firecrawl API.

Provides web connectivity tools for searching the web via DuckDuckGo or SerpAPI, fetching URL content, and extracting readable text from web pages.

Provides eight tools for web search, content extraction, screenshots, PDFs, JavaScript execution, crawling, and Wayback Machine snapshots and archiving.

Enables automated web research and intelligence gathering through recursive web crawling, multi-engine search integration, and persistent SQLite storage with support for keyword filtering and multiple export formats.

Exposes web search, site mapping, scraping, crawling, schema-validated structured extraction, and stateful Playwright browser interaction as MCP tools, routing every call through the same authenticated REST gateway used by the platform's HTTP clients.

Enables web searching and fetching pages converted to Markdown through local stdio MCP tools, so Claude Code and other MCP clients connected via third-party relays retain internet access instead of failing with a server-side tool protocol error. Includes paginated reading of long articles, SSRF/private-network blocking, and charset handling for legacy Chinese sites.

Enables language models to perform AI-powered web scraping, structured data extraction, web search, crawling, schema generation, and monitor management through integrated API tools.

An MCP server enabling AI agents to access the internet via web scraping, crawling, searching, and structured data extraction using Firecrawl.

Provides AI agents and RAG pipelines with production-grade web search, deep research, news, academic, extraction, crawling, and source verification tools. Enables citation-anchored, deduplicated, fresh, and security-hardened evidence gathering from across the web.

A self-hosted MCP server offering unlimited web search, scraping, and content extraction tools with full-content reads, no truncation, and zero per-request cost.

Multi-engine web search and content extraction MCP server with 10 sources, 4-way parallel queries, and content extraction via Jina Reader, no API keys required.

A local MCP server that gives Claude Code web search capabilities using DuckDuckGo, enabling discovery of URLs from natural language queries.

Provides web access tools (fetch, search, sitemap crawl) through String AI's API, automatically handling anti-bot bypass, CAPTCHA, and JavaScript rendering.