"Web scraping and crawling tools" matching MCP servers:

MCP server providing free, IP-direct tools for web search, crawling, anti-detect browsing, and structured data from Wikipedia, Wikidata, GitHub, and YouTube.

Provides web access capabilities for LLMs including search, fetching, content extraction, PDF reading, image viewing, and screenshots.

A token-efficient web browsing, scraping, and crawling MCP server that provides tools for fetching pages as clean markdown or schema-based JSON, verifying content, monitoring changes, and interacting with web pages via a persistent browser pool, complete with SSRF protection and rate limiting.

Comprehensive web research toolkit with 13 tools for searching (via SearXNG), crawling, package discovery, GitHub metrics, error translation, API documentation lookup, data extraction, technology comparison, and service status checking.

Provides local web search and content fetching capabilities for AI assistants, enabling them to search DuckDuckGo and extract clean text from web pages. All requests originate from the user's machine to ensure direct network control and bypass external proxies.

Efficient web scraping MCP server with a two-step workflow for token-efficient content extraction. Supports previewing HTML structure, targeted scraping, and entire documentation site crawling.

A custom MCP server that provides AI agents with tools to fetch web pages, extract readable article text, extract structured data by CSS selector, and check robots.txt permissions.

Enables automated web research and intelligence gathering through recursive web crawling, multi-engine search integration, and persistent SQLite storage with support for keyword filtering and multiple export formats.

Enables web searching and fetching pages converted to Markdown through local stdio MCP tools, so Claude Code and other MCP clients connected via third-party relays retain internet access instead of failing with a server-side tool protocol error. Includes paginated reading of long articles, SSRF/private-network blocking, and charset handling for legacy Chinese sites.

An MCP server for safe web crawling and scraping of public and authenticated sites using Playwright and Crawlee. It provides tools for scraping single pages, crawling entire sites, validating sessions, and listing supported sites, with built-in security protections and adapter-based extensibility.

A local MCP server that gives Claude Code web search capabilities using DuckDuckGo, enabling discovery of URLs from natural language queries.

A web-fetch MCP server for LLM agents that fetches pages with an escalation ladder through different engines, raising FetchBlocked instead of returning blocked content.

Fetches a URL and returns the main content as clean Markdown, using plain HTTP when possible and headless Chromium for JavaScript-rendered or bot-protected pages.

Provides MSP support tools (ticket search, draft response, KB search, update) with a deterministic security guardrail that refuses to draft responses for security tickets based on content scanning, even if mislabeled.

Enables JavaScript-rendered Google searches and readable text extraction from public URLs via a real Chrome instance, along with health and CAPTCHA status monitoring.

Enables Claude Code to delegate web research and image generation to ChatGPT via Playwright browser automation, allowing token-efficient search and visual design iteration without manual screenshots.

EU-native web scraping for AI agents & developers. scrape web pages into markdown, screenshots, metadata, and links, map sites, run asynchronous scrape jobs, and check usage, with EU-hosted, GDPR-aligned processing.

Bridges the editing capabilities of an already-open DingTalk document in the browser to any MCP host via standard stdio, forwarding tools to the page's internal MCP server over a local WebSocket. Supports read/write operations through suggested edits, pairing-based security, and zero runtime dependencies.

Enables Claude to fetch web pages from the user's own local network connection instead of AI datacenter IPs, including optional authenticated sessions, with verdicts for bot challenges, login walls, and other failures.

Local on-demand web search and page reading for coding agents via SearXNG, with SSRF-protected fetching and Markdown extraction.