Web crawler for sequential URL content retrieval MCP servers

"Web crawler for sequential URL content retrieval" matching MCP servers:

• Web Crawler MCP Server

  dhannusch

  -

  security

  F

  license

  -

  quality

  An intelligent web crawling server that uses Cloudflare's headless browser to render dynamic pages and Workers AI to extract relevant links based on natural language queries. It enables AI assistants to search and filter website content while providing secure access through GitHub OAuth authentication.

  Last updated 6 months ago

  1

• Web URL Reader MCP Server

  clsferguson

  -

  security

  F

  license

  -

  quality

  Enables fetching and reading content from web URLs through a configurable proxy prefix. Uses curl to retrieve webpage content and return it as text for analysis or processing.

  Last updated 3 months ago

• Site Crawler MCP

  AndacGuven

  -

  security

  A

  license

  -

  quality

  An MCP server for crawling websites and extracting comprehensive data including images, SEO metadata, security headers, and business intelligence. It features twelve distinct extraction modes to perform detailed audits for e-commerce sites and general web analysis.

  Last updated 6 months ago

  1

  MIT

• content-core

  lfnovo

  A

  security

  A

  license

  A

  quality

  Extract content from URLs, documents, videos, and audio files using intelligent auto-engine selection. Supports web pages, PDFs, Word docs, YouTube transcripts, and more with structured JSON responses.

  Last updated 2 days ago

  1

  118

  MIT

• Documentation Retrieval & Web

  AIwithhassan

  A

  security

  F

  license

  A

  quality

  Enables retrieval and cleaning of official documentation content for popular AI/Python libraries (uv, langchain, openai, llama-index) through web scraping and LLM-powered content extraction. Uses Serper API for search and Groq API to clean HTML into readable text with source attribution.

  Last updated 4 months ago

  1

  1

• MCP-RSS-Crawler

  mshk

  -

  security

  F

  license

  -

  quality

  An MCP server that fetches RSS feeds and shares them with LLMs, allowing AI assistants to access and present the latest news and articles from configured feeds.

  Last updated 10 months ago

  18