"Information about web crawlers or related technology" matching MCP servers:

Enables Codex to send text tasks to ChatGPT web, await responses, continue conversations, choose reasoning effort, and manage projects through MCP tools on macOS.

MCP server providing free, IP-direct tools for web search, crawling, anti-detect browsing, and structured data from Wikipedia, Wikidata, GitHub, and YouTube.

Enables a locally-run LLM to search the web, fetch pages as markdown, make arbitrary HTTP requests, and optionally render pages with headless Chromium.

A self-hosted MCP server providing web search and URL fetching tools, running locally without external API keys or accounts.

Enables Claude and other LLMs to read and parse web content, with smart fallback strategies to bypass access restrictions (e.g., paywalls, Cloudflare) and output Markdown.

Persists web fetch and search results as timestamped JSONL files, indexed for full-text search, enabling users to retrieve and search across archived web pages even after they are removed or changed.

Enables AI agents to drive a logged-in ChatGPT web session over CDP, including managing projects and chats, sending messages with file attachments, and retrieving long Pro replies.

Provides a web search tool with citations by running Claude Code CLI in an isolated Docker container, enabling search even in environments without direct Anthropic web search support.

Enables real-time web search via DeepSeek's search-enhanced dialogue, providing search results as answers through MCP tools.

Comprehensive web research toolkit with 13 tools for searching (via SearXNG), crawling, package discovery, GitHub metrics, error translation, API documentation lookup, data extraction, technology comparison, and service status checking.

A local MCP server for semantic search over RSS/Atom feeds, providing tools for ranked search, trending topics, digests, and feed health monitoring without requiring an API key.

Provides local web search and content fetching capabilities for AI assistants, enabling them to search DuckDuckGo and extract clean text from web pages. All requests originate from the user's machine to ensure direct network control and bypass external proxies.

A reconnaissance MCP server that discovers web data sources by monitoring network requests and extracting API structures, enabling AI to understand data location and format without full crawling.

A local MCP server providing web search, page extraction, and safe browser automation tools for Hermes, Claude Code, and other MCP clients.

Read-only MCP server for WhatsApp Web allowing listing chats and reading recent messages, without sending or mutating WhatsApp state.

Provides web search and page fetch tools for LLM assistants, extracting content and enabling text search within fetched pages.

Enables compatible MCP clients to read JavaScript-rendered web pages as bounded Markdown or structured records, with URL policy and resource limits for safe, inspectable access.

Enables free web searching using Google search results without the need for API keys or authentication. It returns structured data including titles, URLs, and descriptions with configurable result limits.

Automatically crawl and analyze legacy web applications to generate documentation and modernization plans.

Enables users to select UI elements on localhost pages and send fix requests to MCP-compatible coding agents, which retrieve the captures and edit the code, with masking of sensitive values.