"Web crawling techniques and tools" matching MCP servers:

Provides web access capabilities for LLMs including search, fetching, content extraction, PDF reading, image viewing, and screenshots.

Enables AI agents to extract structured content from URLs, search the web via DuckDuckGo, monitor site availability, and convert HTML to markdown without requiring API keys.

Provides local web search and content fetching capabilities for AI assistants, enabling them to search DuckDuckGo and extract clean text from web pages. All requests originate from the user's machine to ensure direct network control and bypass external proxies.

Provides web connectivity tools for searching the web via DuckDuckGo or SerpAPI, fetching URL content, and extracting readable text from web pages.

Enables web searching and fetching pages converted to Markdown through local stdio MCP tools, so Claude Code and other MCP clients connected via third-party relays retain internet access instead of failing with a server-side tool protocol error. Includes paginated reading of long articles, SSRF/private-network blocking, and charset handling for legacy Chinese sites.

Multi-engine web search and content extraction MCP server with 10 sources, 4-way parallel queries, and content extraction via Jina Reader, no API keys required.

MCP server exposing web_search, web_fetch, and cluesift_domain_fetch tools with configurable Ollama API keys and HTTP or stdio transport.

A custom MCP server that provides AI agents with tools to fetch web pages, extract readable article text, extract structured data by CSS selector, and check robots.txt permissions.

An MCP server for safe web crawling and scraping of public and authenticated sites using Playwright and Crawlee. It provides tools for scraping single pages, crawling entire sites, validating sessions, and listing supported sites, with built-in security protections and adapter-based extensibility.

MCP server providing free, IP-direct tools for web search, crawling, anti-detect browsing, and structured data from Wikipedia, Wikidata, GitHub, and YouTube.

A local MCP server providing web search, page extraction, and safe browser automation tools for Hermes, Claude Code, and other MCP clients.

MCP server for multi-engine web search and web page fetching, supporting parallel search, content extraction, and optional LLM-powered search summarization and deep search.

Provides eight tools for web search, content extraction, screenshots, PDFs, JavaScript execution, crawling, and Wayback Machine snapshots and archiving.

A self-hosted MCP server that reads web pages and converts them to Markdown, with built-in web search and anti-crawl fallback using Playwright.

Enables fetching, crawling, and browsing anti-bot protected web pages with real browser rendering, returning clean JSON or markdown, and solving captchas.

MCP server that parses web pages into clean markdown via Crawl4AI's HTTP API, exposing parse_page and health-check tools with configurable proxy and timeout.

A web-fetch MCP server for LLM agents that fetches pages with an escalation ladder through different engines, raising FetchBlocked instead of returning blocked content.

An open-source MCP server providing AI agents with neural web search via Exa and tiered web fetch (Exa, local browser, Firecrawl) as a drop-in replacement for built-in web tools, preserving provenance and guarding against SSRF.

Drop-in replacement for Claude's native web_search and web_fetch tools, resolving IP blocks and user-agent detection by using your own API keys and proxy.

Enables Claude to fetch web pages from the user's own local network connection instead of AI datacenter IPs, including optional authenticated sessions, with verdicts for bot challenges, login walls, and other failures.