"Information about crawer or crawler" matching MCP servers:

Web-crawler MCP server that fetches web pages with static parsing or headless browser fallback, enabling Claude to fetch, extract links, crawl sites, and select content via CSS selectors.

Enables AI assistants to search and collect data from Xiaohongshu/REDnote, including notes, comments, and user info, with dual-layer anti-detection and persistent login.

A specialized web crawler and scraper for the US Patent and Trademark Office (USPTO) website, built with Crawl4AI and MCP integration, enabling patent and trademark search, extraction, and status checking.

Enables site-wide crawling of internal links with BFS traversal, time/title/URL filtering, and page operations like summarization and link clicking, with results persisted to SQLite.

Exposes Futurepedia AI tool crawling and universal web page parsing as MCP tools for assistants.

Enables AI to autonomously search, filter, and download images from the web using natural language commands, with intelligent scoring and local storage.

Enables agents to crawl URLs and retrieve page content as fixed-shape structured JSON, including title, markdown, links, word count, and error status.

A minimal MCP server that crawls web pages using crawl4ai and returns content as markdown, deployable to Cloud Run.

Enables web crawling and content extraction from web pages, supporting multiple output formats like text, markdown, XML, and JSON, with robots.txt compliance and rate limiting.

Provides lightweight web crawling and scraping capabilities, enabling users to fetch pages as clean Markdown, recursively crawl domains, extract metadata, and perform search-and-crawl operations.

An intelligent web crawling server that uses Cloudflare's headless browser to render dynamic pages and Workers AI to extract relevant links based on natural language queries. It enables AI assistants to search and filter website content while providing secure access through GitHub OAuth authentication.

Enables crawling websites, including JavaScript-rendered pages, and returns extracted page content as JSON with structured error reporting.

It crawls website

A single-process MCP server that crawls web pages, handles anti-bot challenges, converts HTML to Markdown, and caches results in PostgreSQL, providing tools to fetch and read web content.

MCP server for crawling and extracting data from web pages using Selenium with CSS/XPath selectors. Supports 17 tools including smart extraction, pagination, infinite scroll, and screenshots.

Autonomous LLM-powered web crawler with 17 browser tools and goal-driven agent. Single Rust binary with stealth browsing, provider-agnostic LLM support, and built-in MCP server.

Enables inspection of any website's AI-search readiness, checking AI crawler blocks, llms.txt, schema markup, and indexing directives from MCP clients like Claude.

Enables web searching and webpage scraping using pure crawler technology without requiring official APIs. Supports Bing web and news search, batch webpage scraping, and content extraction through Puppeteer automation.

An MCP server that allows users to check if a website is experiencing downtime by querying isitdownrightnow.com, providing status information and details about recent downtime events.

Enables web search functionality through Perplexity AI's API with time-based filtering options. Allows users to search the web for recent information within specified time periods (day, week, month, or year).