"Information about crawer or crawler" matching MCP servers:

Web-crawler MCP server that fetches web pages with static parsing or headless browser fallback, enabling Claude to fetch, extract links, crawl sites, and select content via CSS selectors.

Provides live wait times and office details for all 60 Oregon DMV field offices by scraping the ODOT website, with no authentication required.

Enables AI assistants to search and collect data from Xiaohongshu/REDnote, including notes, comments, and user info, with dual-layer anti-detection and persistent login.

An MCP (Model Context Protocol) server that gives AI agents live, structured ad intelligence across Facebook, Google, and Instagram — data that no base model can produce from training alone. Powered by Apify actors. Works with any MCP-compatible client: Cursor, Claude, etc.

Enables comprehensive website SEO analysis including crawling, on-page audits, site structure visualization, and content topic classification with taxonomy mapping.

This server crawls documentation sites and provides search and page-retrieval tools through MCP, enabling users to query indexed documentation content.

Enables site-wide crawling of internal links with BFS traversal, time/title/URL filtering, and page operations like summarization and link clicking, with results persisted to SQLite.

A comprehensive website crawler and SEO analyzer that stores site data in a local SQLite database for AI-driven auditing. It enables users to detect technical SEO issues, broken links, and security vulnerabilities through natural language queries or terminal commands.

Exposes Futurepedia AI tool crawling and universal web page parsing as MCP tools for assistants.

An MCP server for crawling websites and extracting comprehensive data including images, SEO metadata, security headers, and business intelligence. It features twelve distinct extraction modes to perform detailed audits for e-commerce sites and general web analysis.

Enables AI to autonomously search, filter, and download images from the web using natural language commands, with intelligent scoring and local storage.

Enables agents to crawl URLs and retrieve page content as fixed-shape structured JSON, including title, markdown, links, word count, and error status.

A specialized web crawler and scraper for the US Patent and Trademark Office (USPTO) website, built with Crawl4AI and MCP integration, enabling patent and trademark search, extraction, and status checking.

A minimal MCP server that crawls web pages using crawl4ai and returns content as markdown, deployable to Cloud Run.

Enables keyword, creator, or item-based image and video retrieval and downloading from Douyin, Xiaohongshu, Weibo, Bilibili, X, and Instagram. It adds intent parsing, semantic and visual re-ranking, quality filtering, and deduplication so results are relevant rather than every image on a page.

Enables web crawling and content extraction from web pages, supporting multiple output formats like text, markdown, XML, and JSON, with robots.txt compliance and rate limiting.

Provides lightweight web crawling and scraping capabilities, enabling users to fetch pages as clean Markdown, recursively crawl domains, extract metadata, and perform search-and-crawl operations.

Enables AI assistants such as Claude, ChatGPT, Cursor, and Claude Code to run 25+ specialized SEO checks on any URL, generate XML sitemaps and robots.txt files, extract brand design elements, and audit technical SEO, content quality, mobile responsiveness, and security.

Enables crawling news articles by category (politics, economy, society, etc.) and generates blog-ready markdown summaries. Provides MCP tools for listing categories, crawling news, getting article details, and saving to files.

An intelligent web crawling server that uses Cloudflare's headless browser to render dynamic pages and Workers AI to extract relevant links based on natural language queries. It enables AI assistants to search and filter website content while providing secure access through GitHub OAuth authentication.

Enables crawling websites, including JavaScript-rendered pages, and returns extracted page content as JSON with structured error reporting.

Enables autonomous web browsing and extraction by canonicalizing URLs, detecting anti-crawler traps such as circular paths and infinite pagination, pruning DOM content, mapping forms, converting tables to JSON, and evaluating agent trajectories.

Enables autonomous web agents to canonicalize URLs, detect anti-crawler traps such as circular paths and infinite pagination, prune DOM tokens, auto-map forms, transform tables into JSON, and evaluate action trajectories through MCP.

It crawls website

Crawl entire websites, sitemaps, and documentation portals into clean, LLM-ready Markdown without ads, cookie popups, or noise for RAG pipelines and AI agents.

Free, open-source desktop SEO crawler with a built-in MCP server. 250 technical SEO, GEO and AI-search checks across 23 areas, run locally; MCP tools list crawls, inspect issues, run Lighthouse, check AI-crawler access, and apply confirmation-gated WordPress fixes with rollback.

Enables agents to read any web page or document as markdown, html, links, screenshots or extracted JSON, map and crawl entire sites, and drive a persistent browser session that stays open between calls so pages can be inspected and clicked without reloading. Callers pay per request with a wallet over x402 — no API key or signup — and every route answers its own free GET with the fields it expects.

Toolset that crawls websites, generates Markdown documentation, and makes that documentation searchable via a Model Context Protocol (MCP) server for integration with tools like Cursor.

Enables AI clients to query financial news from Yahoo Finance, CNBC, configurable RSS feeds, and 鉅亨網 (cnYES), and to fetch historical OHLCV market data from Yahoo and Investing.com. Supports stock and keyword search, category subscriptions, time filtering, article body extraction, and stdio integration with Cursor, Claude Desktop, and Spring AI.

Lets an MCP client read allowlisted pages from the LiveSwapFace website, retrieve its official links, and estimate credits for a planned browser session using rates fetched at call time. It is read-only and never performs face swaps, signs in, uploads media, or touches a private API.