"Information about web crawlers or scraping techniques" matching MCP servers:

MCP server that provides OpenRouter model pricing data, enabling price lookups, trending/cheapest lists, and model searches without an API key.

An MCP (Model Context Protocol) server that gives AI agents live, structured ad intelligence across Facebook, Google, and Instagram — data that no base model can produce from training alone. Powered by Apify actors. Works with any MCP-compatible client: Cursor, Claude, etc.

HandaaS的MCP服务器，提供全域资讯搜索、企业新闻、行业资讯、主题跟踪和企业动态监控能力，帮助用户开展行业研究和舆情分析。

Enables web searches through DuckDuckGo without an API key and extracts readable main content from web pages.

Enables web search and web fetch operations using Ollama's hosted APIs, allowing MCP clients to search the web and retrieve page content.

MCP server for web page fetching (converting to Markdown/text with automatic fallback between Tavily and Firecrawl) and web search via Tavily.

Enables AI agents to search and fetch high-quality information from multiple sources, including general web APIs, Wikipedia, arXiv, Hacker News, Stack Exchange, and Crossref, with optional pro-mode deep research and clean markdown page extraction.

Enables a locally-run LLM to search the web, fetch pages as markdown, make arbitrary HTTP requests, and optionally render pages with headless Chromium.

A self-hosted MCP server providing web search and URL fetching tools, running locally without external API keys or accounts.

Persists web fetch and search results as timestamped JSONL files, indexed for full-text search, enabling users to retrieve and search across archived web pages even after they are removed or changed.

Provides a web search tool with citations by running Claude Code CLI in an isolated Docker container, enabling search even in environments without direct Anthropic web search support.

A local MCP server for semantic search over RSS/Atom feeds, providing tools for ranked search, trending topics, digests, and feed health monitoring without requiring an API key.

Provides local web search and content fetching capabilities for AI assistants, enabling them to search DuckDuckGo and extract clean text from web pages. All requests originate from the user's machine to ensure direct network control and bypass external proxies.

A local MCP server providing web search, page extraction, and safe browser automation tools for Hermes, Claude Code, and other MCP clients.

Enables web searching via DuckDuckGo and extracting readable content from any URL using Mozilla Readability, providing web context similar to Cursor's built-in functionality.

A comprehensive web scraping server that transforms web content into clean, agent-ready Markdown with automatic citations and efficient caching. It features a robust suite of tools for metadata extraction, sentiment analysis, SEO auditing, and security scanning while strictly adhering to robots.txt policies.

A powerful tool for fetching and extracting text content from web pages and APIs, supporting web scraping, REST API requests, and Google Custom Search integration.

Provides web connectivity tools for searching the web via DuckDuckGo or SerpAPI, fetching URL content, and extracting readable text from web pages.

Enables web research through multi-provider search, documentation lookup, public code search, and clean Markdown extraction from static or JavaScript-rendered pages via five read-only tools.

MCP server for multi-engine web search and web page fetching, supporting parallel search, content extraction, and optional LLM-powered search summarization and deep search.