"Web crawler for sequential URL content retrieval" matching MCP servers:

Enables AI agents to fetch, crawl, extract, and search web content through a unified set of tools, with automatic routing to specialized backends.

An open-source MCP server providing AI agents with neural web search via Exa and tiered web fetch (Exa, local browser, Firecrawl) as a drop-in replacement for built-in web tools, preserving provenance and guarding against SSRF.

Enables MCP-capable agents to perform reliable web search and content fetching through a single server that automatically fails over across multiple providers. It handles keyed or keyless authentication automatically and returns clean Markdown or text results.

An intelligent web crawling server that uses Cloudflare's headless browser to render dynamic pages and Workers AI to extract relevant links based on natural language queries. It enables AI assistants to search and filter website content while providing secure access through GitHub OAuth authentication.

Enables fetching and reading content from web URLs through a configurable proxy prefix. Uses curl to retrieve webpage content and return it as text for analysis or processing.

An MCP server providing web search, image search, and page scraping tools to LLMs without requiring API keys.

MCP server that gives AI agents access to current web information through web_search and fetch_url tools, using Tavily, Brave, and DuckDuckGo providers with automatic fallback.

Enables AI agents to parse PDFs and search local document context through multimodal extraction and vector-based retrieval, all exposed as MCP tools.

This server crawls documentation sites and provides search and page-retrieval tools through MCP, enabling users to query indexed documentation content.

A comprehensive website crawler and SEO analyzer that stores site data in a local SQLite database for AI-driven auditing. It enables users to detect technical SEO issues, broken links, and security vulnerabilities through natural language queries or terminal commands.

A Model Context Protocol server that enables LLMs to fetch and process web content in multiple formats (HTML, JSON, Markdown, text) with automatic format detection.

An MCP server for crawling websites and extracting comprehensive data including images, SEO metadata, security headers, and business intelligence. It features twelve distinct extraction modes to perform detailed audits for e-commerce sites and general web analysis.

An evaluation harness that probes MCP-based retrieval servers for calibration, relevance, coverage, citation integrity, and more, also exposing the probes as MCP tools for assistants and CI agents.

SERP-based content scoring and optimization with 7 SEO categories.

Enables AI agents to search private GitHub Pages by authenticating in a dedicated browser and retrieving indexed sections from a local SQLite database, with support for multiple sites and languages.

Provides Baidu hot search and Bilibili trending video data for Claude and Cherry Studio, supporting retrieval, search, and cache management.

Enables searching and retrieving images from Wikipedia Commons with detailed metadata including licenses and authors.

Fetches and tracks content from YouTube channels, RSS feeds, and GitHub releases with persistence to identify new items across sessions.

The URL-Context-MCP MCP Server provides a tool to analyze and summarize the content of URLs using Google Gemini's URL Context capability via the Gemini API. Now also supports optional grounding with Google Search alongside URL Context. The server is designed to follow prompt-only orchestration: con

MCP server for multi-engine web search and web page fetching, supporting parallel search, content extraction, and optional LLM-powered search summarization and deep search.