"Web scraping and crawling tools" matching MCP servers:

Provides web access capabilities for LLMs including search, fetching, content extraction, PDF reading, image viewing, and screenshots.

Provides an LLM client with self-hosted web scraping capabilities including scraping, crawling, mapping, searching, structured data extraction, browser interaction, and deep research, without reliance on paid proxy or CAPTCHA services.

Web search, content extraction, and library docs for AI agents with 5-strategy scraping and runs without API keys.

Comprehensive web research toolkit with 13 tools for searching (via SearXNG), crawling, package discovery, GitHub metrics, error translation, API documentation lookup, data extraction, technology comparison, and service status checking.

MCP server exposing web_search, web_fetch, and cluesift_domain_fetch tools with configurable Ollama API keys and HTTP or stdio transport.

Provides browser automation and web scraping as MCP tools, enabling autonomous URL ingestion, crawling, extraction, and anti-bot handling with interactive browser control.

A local MCP server that gives Claude Code web search capabilities using DuckDuckGo, enabling discovery of URLs from natural language queries.

Provides MCP tools for web scraping, including smart fetching, proxy rotation, and browser rendering to bypass anti-bot challenges and render JavaScript-heavy sites.

MCP server that parses web pages into clean markdown via Crawl4AI's HTTP API, exposing parse_page and health-check tools with configurable proxy and timeout.

A web-fetch MCP server for LLM agents that fetches pages with an escalation ladder through different engines, raising FetchBlocked instead of returning blocked content.

Fetches a URL and returns the main content as clean Markdown, using plain HTTP when possible and headless Chromium for JavaScript-rendered or bot-protected pages.

An open-source MCP server providing AI agents with neural web search via Exa and tiered web fetch (Exa, local browser, Firecrawl) as a drop-in replacement for built-in web tools, preserving provenance and guarding against SSRF.

Provides MSP support tools (ticket search, draft response, KB search, update) with a deterministic security guardrail that refuses to draft responses for security tickets based on content scanning, even if mislabeled.

Browser automation that bypasses anti-bot systems and handles web-scraping with CDP Mode.

Enables Claude Code to delegate web research and image generation to ChatGPT via Playwright browser automation, allowing token-efficient search and visual design iteration without manual screenshots.

Connect AI assistants to your Gramps Web family tree. Search and edit genealogy records, explore ancestors, descendants, relationships and timelines, and read document scans with vision-capable models. Includes guided research workflows, source citations, read-only mode, and privacy-aware media previews. Install with Docker or a Claude Desktop bundle.

Bridges the editing capabilities of an already-open DingTalk document in the browser to any MCP host via standard stdio, forwarding tools to the page's internal MCP server over a local WebSocket. Supports read/write operations through suggested edits, pairing-based security, and zero runtime dependencies.

Enables Claude to fetch web pages from the user's own local network connection instead of AI datacenter IPs, including optional authenticated sessions, with verdicts for bot challenges, login walls, and other failures.

Open-source browser automation API with anti-detection for AI agents, web scraping, and automation, providing undetectable Chrome via MCP and REST API with Cloudflare bypass.

Ad-free web scraping and search exposed as 3 MCP tools. fetch_page, fetch_pages_batch, search_web. Works with Claude Desktop, Cursor, Cline.