"Information about web crawlers or related technology" matching MCP servers:

MCP server that provides OpenRouter model pricing data, enabling price lookups, trending/cheapest lists, and model searches without an API key.

An MCP (Model Context Protocol) server that gives AI agents live, structured ad intelligence across Facebook, Google, and Instagram — data that no base model can produce from training alone. Powered by Apify actors. Works with any MCP-compatible client: Cursor, Claude, etc.

HandaaS的MCP服务器，提供全域资讯搜索、企业新闻、行业资讯、主题跟踪和企业动态监控能力，帮助用户开展行业研究和舆情分析。

A minimal MCP server that crawls web pages using crawl4ai and returns content as markdown, deployable to Cloud Run.

Provides current weather information for any location and generates AI-powered responses using Google Gemini through FastMCP.

Enables web search and web fetch operations using Ollama's hosted APIs, allowing MCP clients to search the web and retrieve page content.

Enables Codex to send text tasks to ChatGPT web, await responses, continue conversations, choose reasoning effort, and manage projects through MCP tools on macOS.

MCP server providing free, IP-direct tools for web search, crawling, anti-detect browsing, and structured data from Wikipedia, Wikidata, GitHub, and YouTube.

Enables AI agents to search and fetch high-quality information from multiple sources, including general web APIs, Wikipedia, arXiv, Hacker News, Stack Exchange, and Crossref, with optional pro-mode deep research and clean markdown page extraction.

Enables a locally-run LLM to search the web, fetch pages as markdown, make arbitrary HTTP requests, and optionally render pages with headless Chromium.

Persists web fetch and search results as timestamped JSONL files, indexed for full-text search, enabling users to retrieve and search across archived web pages even after they are removed or changed.

MCP server for web scraping with natural language, enabling site crawling, page fetching, and extraction of emails, phones, links, and social profiles. Returns clean markdown, handles JavaScript rendering, and respects rate limits.

Enables AI agents to drive a logged-in ChatGPT web session over CDP, including managing projects and chats, sending messages with file attachments, and retrieving long Pro replies.

Comprehensive web research toolkit with 13 tools for searching (via SearXNG), crawling, package discovery, GitHub metrics, error translation, API documentation lookup, data extraction, technology comparison, and service status checking.

A local MCP server for semantic search over RSS/Atom feeds, providing tools for ranked search, trending topics, digests, and feed health monitoring without requiring an API key.

A reconnaissance MCP server that discovers web data sources by monitoring network requests and extracting API structures, enabling AI to understand data location and format without full crawling.

A local MCP server providing web search, page extraction, and safe browser automation tools for Hermes, Claude Code, and other MCP clients.

A headless web scraping server that extracts main content from web pages into Markdown, text, or HTML for AI and automation integration. It features per-domain rate limiting and robust error handling using Playwright and BeautifulSoup.

Provides web search and content retrieval optimized for AI coding assistants, returning full conversations and structured content from StackOverflow, GitHub Issues, arXiv, and Wikipedia in a single call.

A comprehensive web scraping MCP server with 26 tools for fetching, parsing, extracting, and assisting with web content, returning Markdown-formatted results.