"A query related to 'crawl' and its contexts" matching MCP servers:

MCP server to crawl WeChat articles, download images, and convert to Markdown, designed for Cursor IDE and AI tools.

MCP server for Cloudflare Browser Rendering Crawl API. Fetches and crawls web pages, returning clean Markdown optimized for LLM consumption.

MCP server for web crawling, searching, and AI-powered content extraction, supporting single-page, batch, and full-site crawling along with text, news, image, book, and video search.

Enables LLMs to crawl websites via Cloudflare's Browser Rendering API, providing tools to start, monitor, cancel crawls, and retrieve structured content with AI extraction.

Provides preflight checks and domain intelligence so agents can skip domains that will refuse them or charge a fee, covering robots.txt, edge refusals, and HTTP 402 paywalls.

Enables fetching, crawling, and analyzing web pages with 29 tools for SEO audits, content extraction, and more.

Enables converting webpages into structured JSON using CSS-selector schemas, with anti-bot handling and support for protected pages.

Enables converting any webpage to clean LLM-ready markdown, with extraction, article, and news modes, using a real browser with anti-bot handling.

Fetches a URL and returns the main content as clean Markdown, using plain HTTP when possible and headless Chromium for JavaScript-rendered or bot-protected pages.

Enables AI agents to fetch one or many web pages concurrently and receive clean, LLM-ready Markdown with navigation, ads and boilerplate stripped away, plus extracted metadata such as title, description, author, Open Graph tags and canonical URL. It runs locally over stdio with no API key or account, requiring only a single command.

Fetches website content and converts it to Markdown format with AI-powered content cleanup, ad removal, and full OpenAPI/Swagger specification support for easy processing by AI assistants.

Converts X (Twitter) posts into JSON, PDF, PNG, Markdown, or Slack/Discord messages using public endpoints, no API key or login required.

Enables AI agents to control a Playwright-driven browser through natural-language instructions, including page interaction and live browser task execution.

MCP server that fetches remote or local HTML resources and converts them to Markdown, enabling AI clients to summarize or process web content.

Enables AI agents to convert HTML content into clean Markdown, stripping scripts and styles while preserving headings, links, lists, and images, with pay-per-call x402 micropayments and no API keys.

An MCP server for web content extraction that converts HTML pages into clean, LLM-optimized Markdown using Mozilla's Readability. It supports batch processing, intelligent multi-page crawling, and configurable caching while respecting robots.txt standards.

Enables AI assistants to search for vehicles, manage reservations, and access Enterprise Plus loyalty account information through headless browser automation on enterprise.com. It supports the complete car rental workflow including vehicle searching, booking, modifications, and points balance checks.

Self-hosted web-intelligence server for Hermes Agent and other MCP clients, featuring hybrid HTTP+browser crawling, document parsing, search, structured extraction, and change detection.

Enables agents to transform tabular Markdown content into structured, type-inferred records with normalized formatting through an MCP-compatible interface.

Enables converting Markdown and HTML tables into structured JSON records with automatic column type inference and format normalization. It also offers supporting web-agent primitives such as DOM semantic pruning, form field auto-mapping, and execution trajectory evaluation, all via a zero-dependency MCP server.