"A search for information about deep crawl web crawling and site auditing tools" matching MCP servers:

MCP server for web crawling, searching, and AI-powered content extraction, supporting single-page, batch, and full-site crawling along with text, news, image, book, and video search.

A local MCP server that gives Claude Code web search capabilities using DuckDuckGo, enabling discovery of URLs from natural language queries.

Enables web search and web fetch operations using Ollama's hosted APIs, allowing MCP clients to search the web and retrieve page content.

Provides a unified web search tool for AI agents by running multiple search providers concurrently and returning normalized results. It integrates with Claude Code, Codex CLI, OpenCode, and Hermes.

Provides web access capabilities for LLMs including search, fetching, content extraction, PDF reading, image viewing, and screenshots.

Provides DuckDuckGo web search and site fetching tools for Claude Desktop, enabling up to 10 search results and text content extraction from URLs.

Provides a web search tool with citations by running Claude Code CLI in an isolated Docker container, enabling search even in environments without direct Anthropic web search support.

Provides local web search and content fetching capabilities for AI assistants, enabling them to search DuckDuckGo and extract clean text from web pages. All requests originate from the user's machine to ensure direct network control and bypass external proxies.

Unofficial MCP server wrapping crawl4ai that enables extraction and analysis of content from web pages, PDFs, Office documents, YouTube videos, and more, with AI-powered summarization and Google search integration to reduce token usage while preserving key information.

An experimental MCP search server for Claude Code. Returns raw, verbatim web content instead of AI-rewritten summaries.

A high-performance search service that converts results from Google, Bing, and DuckDuckGo into structured JSON or Markdown. It features multi-layer depth crawling and uses the Camoufox anti-detection browser for reliable content extraction and fallback search logic.

Enables AI agents to run local multi-engine web searches, fetch pages through anti-bot fallback chains, and query site-specific sources such as GitHub, npm, PyPI, Docker Hub, and Twitter/X for structured results.

Provides free web search, content fetching, image search, and deep research via SearXNG, no API keys required.

Provides AI agents and RAG pipelines with production-grade web search, deep research, news, academic, extraction, crawling, and source verification tools. Enables citation-anchored, deduplicated, fresh, and security-hardened evidence gathering from across the web.

Provides 19 tools for web search, content fetching, and extraction, including DuckDuckGo search, YouTube transcripts, and webpage screenshots.

Provides a free web search tool to MCP clients by connecting to a self-hosted SearXNG metasearch instance, enabling users to search the web with customizable parameters like categories and time range.

Multi-engine web search and content extraction MCP server with 10 sources, 4-way parallel queries, and content extraction via Jina Reader, no API keys required.

MCP server for Cloudflare Browser Rendering Crawl API. Fetches and crawls web pages, returning clean Markdown optimized for LLM consumption.

Site, email and agent safety — SPF/DKIM/DMARC, AI/SEO audit, page-change monitor, free hidden-AI scanner (4 MCP tools).

A lean FastMCP server that wraps xAI's web_search and x_search tools for MCP hosts, enabling live web and X search with citations.

Enables MCP-compatible agents to perform web searches via the agent-web-search engine, returning ranked results with sources.

Provides LLMs with real-time web search and content extraction capabilities, including text/news search, full-text URL reading, and targeted technical documentation search.

Enables LLMs to crawl websites via Cloudflare's Browser Rendering API, providing tools to start, monitor, cancel crawls, and retrieve structured content with AI extraction.

MCP server and CLI for querying Common Crawl via AWS Athena, enabling users to check a domain's presence in crawls, analyze dynamics, compare domains, and inspect page text.

Enables fetching and extracting text from URLs, searching the web via DuckDuckGo and Yandex, and retrieving page links through MCP tools.

Turns any website URL into a crawled markdown corpus and knowledge graph, then exposes it to AI agents over MCP. Enables full-text search, full page retrieval, subgraph neighbor expansion, and site statistics so agents can answer questions grounded in a specific site's documentation.

An extended, self-hosted web research server for MCP-compatible clients that provides tools for web search, page fetching, JavaScript rendering, Internet Archive snapshots, and YouTube transcripts.

Enables web search across multiple search engines (DuckDuckGo, Bing, Startpage) with parallel execution and result deduplication. Also provides web page content extraction capabilities.

Enables MCP clients to conduct search-first web research through source discovery, page retrieval by ID, and direct URL reading.

Enables web searching through Google, DuckDuckGo, and Bing using a headless Chrome browser, returning structured results with titles, URLs, and snippets. Also supports fetching and extracting text content from any webpage.