information-about-web-crawling-or-spiders MCP servers

"information-about-web-crawling-or-spiders" matching MCP servers:

• Web-Scout

  UpperMoon0

  -

  security

  F

  license

  -

  quality

  Enables AI-powered web searches using DuckDuckGo with automatic summarization via Google's Gemini AI. Supports both summary and detailed analysis modes for search results.

  Last updated 13 days ago

• mcp-web

  TsFreddie

  -

  security

  A

  license

  -

  quality

  Enables AI agents to search DuckDuckGo and fetch web content with human-assisted CAPTCHA handling, designed for local use where humans can intervene when bot detection occurs.

  Last updated 4 months ago

  39

  MIT

• Web Research Assistant

  elad12390

  A

  security

  A

  license

  A

  quality

  Comprehensive web research toolkit with 13 tools for searching (via SearXNG), crawling, package discovery, GitHub metrics, error translation, API documentation lookup, data extraction, technology comparison, and service status checking.

  Last updated 2 months ago

  13

  4

  MIT

• Web Analysis MCP

  himly0302

  -

  security

  F

  license

  -

  quality

  Enables intelligent web searching using SearXNG with content crawling via Creeper, then summarizes webpage content using LLM to avoid token limit issues. Supports smart filtering with domain blacklist/whitelist and optional LLM-based relevance filtering.

  Last updated a month ago

• Web Search MCP

  uk0

  -

  security

  F

  license

  -

  quality

  A high-performance search service that converts results from Google, Bing, and DuckDuckGo into structured JSON or Markdown. It features multi-layer depth crawling and uses the Camoufox anti-detection browser for reliable content extraction and fallback search logic.

  Last updated 11 days ago

  13

• web-scrapper-stdio

  JustAzul

  -

  security

  A

  license

  -

  quality

  A headless web scraping server that extracts main content from web pages into Markdown, text, or HTML for AI and automation integration. It features per-domain rate limiting and robust error handling using Playwright and BeautifulSoup.

  Last updated 6 months ago

  MIT