"A query related to 'crawl' and its contexts" matching MCP servers:

MCP server to crawl WeChat articles, download images, and convert to Markdown, designed for Cursor IDE and AI tools.

MCP server for Cloudflare Browser Rendering Crawl API. Fetches and crawls web pages, returning clean Markdown optimized for LLM consumption.

MCP server for web crawling, searching, and AI-powered content extraction, supporting single-page, batch, and full-site crawling along with text, news, image, book, and video search.

Enables LLMs to crawl websites via Cloudflare's Browser Rendering API, providing tools to start, monitor, cancel crawls, and retrieve structured content with AI extraction.

Unofficial MCP server wrapping crawl4ai that enables extraction and analysis of content from web pages, PDFs, Office documents, YouTube videos, and more, with AI-powered summarization and Google search integration to reduce token usage while preserving key information.

Provides preflight checks and domain intelligence so agents can skip domains that will refuse them or charge a fee, covering robots.txt, edge refusals, and HTTP 402 paywalls.

MCP server for Common Crawl CDX that enables backlink discovery, expired domain finding, and competitor gap analysis without requiring API keys.

Checks a website's AI readiness: can ChatGPT, Claude, Perplexity, Google AI and 50+ other AI crawlers (GPTBot, ClaudeBot, PerplexityBot) access it? Scores the site, then generates llms.txt, an AI-crawler robots.txt and JSON-LD, and reads LLM Monitor trends, from Claude Desktop, Cursor and other MCP clients.

Enables fetching, crawling, and analyzing web pages with 29 tools for SEO audits, content extraction, and more.

Crawls Korean stock market data from Naver Finance, providing real-time information on top searched stocks and detailed stock information by stock code.

MCP server that fetches remote or local HTML resources and converts them to Markdown, enabling AI clients to summarize or process web content.

An MCP server that automates converting diverse content sources like WeChat articles, YouTube videos, and various document formats into AI-generated outputs such as podcasts and slide decks via Google NotebookLM. It integrates specialized tools for web scraping, OCR, and file transformation to facilitate seamless content generation through natural language.

Enables converting webpages into structured JSON using CSS-selector schemas, with anti-bot handling and support for protected pages.

Enables converting any webpage to clean LLM-ready markdown, with extraction, article, and news modes, using a real browser with anti-bot handling.

Fetches a URL and returns the main content as clean Markdown, using plain HTTP when possible and headless Chromium for JavaScript-rendered or bot-protected pages.

Enables converting a public release-note URL into ordered upgrade tasks with source excerpts, anchors, confidence, and uncertainty, without executing release text or project code.

Enables AI agents to fetch one or many web pages concurrently and receive clean, LLM-ready Markdown with navigation, ads and boilerplate stripped away, plus extracted metadata such as title, description, author, Open Graph tags and canonical URL. It runs locally over stdio with no API key or account, requiring only a single command.

Fetches website content and converts it to Markdown format with AI-powered content cleanup, ad removal, and full OpenAPI/Swagger specification support for easy processing by AI assistants.

Converts X (Twitter) posts into JSON, PDF, PNG, Markdown, or Slack/Discord messages using public endpoints, no API key or login required.

An MCP server for web content extraction that converts HTML pages into clean, LLM-optimized Markdown using Mozilla's Readability. It supports batch processing, intelligent multi-page crawling, and configurable caching while respecting robots.txt standards.