"A server for crawling web pages and finding subpages" matching MCP servers:

MCP server that gives AI agents access to current web information through web_search and fetch_url tools, using Tavily, Brave, and DuckDuckGo providers with automatic fallback.

Provides a local MCP server for AI-powered web crawling on Windows, enabling agents to directly crawl and extract web content without requiring WSL.

A self-hosted MCP server that reads web pages and converts them to Markdown, with built-in web search and anti-crawl fallback using Playwright.

An MCP server for safe web crawling and scraping of public and authenticated sites using Playwright and Crawlee. It provides tools for scraping single pages, crawling entire sites, validating sessions, and listing supported sites, with built-in security protections and adapter-based extensibility.

A fast, dependency-light MCP server that converts web pages into clean Markdown, structured metadata, and classified links for language models.

A fault-tolerant, stealth-enabled Model Context Protocol (MCP) server for web searching and content fetching. Built for AI Agents (Cursor, Claude Code, OpenCode), it uses a stealth browser engine to fetch pages, dynamically handles SPAs/React, and converts bloat into token-optimized Markdown.

MCP server providing free, IP-direct tools for web search, crawling, anti-detect browsing, and structured data from Wikipedia, Wikidata, GitHub, and YouTube.

Enables a locally-run LLM to search the web, fetch pages as markdown, make arbitrary HTTP requests, and optionally render pages with headless Chromium.

Enables LLMs to fetch and process web page contents by providing tools to retrieve raw HTML, extract clean text, and convert web pages to markdown format.

Enables AI assistants to inspect, debug, and test web pages using Playwright. Provides comprehensive DOM inspection, visibility debugging, layout validation, and element finding capabilities in real browser environments.

A local MCP server providing web search, page extraction, and safe browser automation tools for Hermes, Claude Code, and other MCP clients.

Enables web browsing capabilities using BeautifulSoup4

Provides web search and page fetch tools for LLM assistants, extracting content and enabling text search within fetched pages.

A powerful tool for fetching and extracting text content from web pages and APIs, supporting web scraping, REST API requests, and Google Custom Search integration.

Provides web connectivity tools for searching the web via DuckDuckGo or SerpAPI, fetching URL content, and extracting readable text from web pages.

Enables web research through multi-provider search, documentation lookup, public code search, and clean Markdown extraction from static or JavaScript-rendered pages via five read-only tools.

Provides lightweight web crawling and scraping capabilities, enabling users to fetch pages as clean Markdown, recursively crawl domains, extract metadata, and perform search-and-crawl operations.

Enables AI assistants to inspect web pages, monitor network requests, extract HTML, analyze console output, and examine DOM elements in real-time through a Playwright-powered browser.

MCP server for multi-engine web search and web page fetching, supporting parallel search, content extraction, and optional LLM-powered search summarization and deep search.

A headless web scraping server that extracts main content from web pages into Markdown, text, or HTML for AI and automation integration. It features per-domain rate limiting and robust error handling using Playwright and BeautifulSoup.