"Understanding Web Scrubbing Techniques" matching MCP servers:

A self-hosted MCP server that reads web pages and converts them to Markdown, with built-in web search and anti-crawl fallback using Playwright.

MCP server that parses web pages into clean markdown via Crawl4AI's HTTP API, exposing parse_page and health-check tools with configurable proxy and timeout.

A web-fetch MCP server for LLM agents that fetches pages with an escalation ladder through different engines, raising FetchBlocked instead of returning blocked content.

Enables Claude to fetch web pages from the user's own local network connection instead of AI datacenter IPs, including optional authenticated sessions, with verdicts for bot challenges, login walls, and other failures.

Enables web searches through DuckDuckGo without an API key and extracts readable main content from web pages.

Enables agents to search the live web and fetch URLs as clean Markdown, returning ranked search results and paginated page content through two lightweight tools.

Wraps Xiaomi MiMo's web_search API into an MCP tool for Claude Code, enabling real-time web searches with location awareness, automatic retry, and timeout control.

Enables Codex to send text tasks to ChatGPT web, await responses, continue conversations, choose reasoning effort, and manage projects through MCP tools on macOS.

Enables reading Slack channels, messages, and threads using the web interface via a persistent browser session, designed for guest users without API access. Does not send messages.

MCP server for AI-powered web research — search (Google, Brave, Serper, SearXNG), scrape any page, extract PDFs/DOCX/YouTube transcripts, academic & patent search. Single Go binary. Works with Claude, Cursor, Copilot, and any MCP client.

Provides LLMs with real-time web search and content extraction capabilities, including text/news search, full-text URL reading, and targeted technical documentation search.

Enables reading web pages and PDFs as clean Markdown, extracting main content and letting agents query for relevant passages, outline sections, search in-page, and list links while respecting robots.txt and blocking private addresses.

The MCP for the Web Speed Agent SDK that enables post-auth agents.

Enables private web search and webpage content extraction using a local SearxNG instance, prioritizing user privacy and autonomy.

A DOM-first, vision-second MCP server for webpage analysis, combining structural DOM extraction with MiniMax vision to provide accurate, context-rich page understanding for AI coding assistants.

MCP server that persists web fetch and search results as timestamped JSONL, enabling long-term archival and cross-domain search via unified-history-mcp.

Enables live-web research by providing stateless browsing, link extraction, and page search capabilities via Playwright, allowing agents to fetch and analyze web content without session management.

MCP server that lets AI editors execute JavaScript, read console output, and simulate clicks/input on static web pages. Supports multiple browser pages and can be deployed remotely via HTTP transport.

MCP server providing free, IP-direct tools for web search, crawling, anti-detect browsing, and structured data from Wikipedia, Wikidata, GitHub, and YouTube.

A local-first MCP server for web search and documentation fetching, providing free DuckDuckGo search, URL-to-markdown conversion, and library docs from npm, PyPI, crates.io, Go, and GitHub with caching.