A web crawler for collecting and indexing website content

Search for:

A web crawler for collecting and indexing website content

View all MCP Servers

Why this server?
A strong general-purpose site crawler for extracting links, tables, lists, and file downloads with crawl depth and page-limit controls.
Site Crawler for Links and
Web Scraping Browser Automation
mambalabsdev
A
license
A
quality
B
maintenance
Enables crawling sites from start URLs and retrieving internal and external links, file download links, HTML tables as rows, and lists, with controls for crawl depth, page limits, sitemaps, robots.txt, and proxy mode.
Updated 3 days ago (2026-11-11 09:10 UTC)
1
728 npm
MIT
Why this server?
A web-crawler MCP server that fetches pages, extracts links, crawls sites, and supports CSS selector content extraction.
crawler-mcp
Browser Automation Web Scraping
shadab15github
A
license
A
quality
D
maintenance
Web-crawler MCP server that fetches web pages with static parsing or headless browser fallback, enabling Claude to fetch, extract links, crawl sites, and select content via CSS selectors.
Updated 3 months ago (2026-07-31 15:47 UTC)
4
ISC
Why this server?
A documentation crawler that indexes documentation sites so MCP-compatible AI clients can search and read crawled content.
rock-mcp
Documentation Access Search RAG Systems
sethwebster
F
license
Not graded
quality
D
maintenance
Documentation crawler MCP server that crawls and indexes documentation sites so that any MCP-compatible AI can search, read, and expand on the content.
Updated 9 months ago (2026-02-21 20:41 UTC)
1
-
Why this server?
A real-time web search and crawling API designed for LLM applications, RAG pipelines, and AI agents.
crawleo-mcp
Browser Automation Search Web Scraping
Crawleo
A
license
B
quality
D
maintenance
Crawleo is a privacy-first, real-time web search and crawling API built for LLM applications, RAG pipelines, AI agents, and automation workflows. With a single API call, Crawleo performs live web search, optionally crawls result pages, and returns clean, AI-ready data in formats like Markdown, Text
Updated 10 months ago (2025-12-31 16:13 UTC)
2
16 npm
11
MIT
Why this server?
An open-source technical SEO crawler MCP server that runs full site audits and supports large-site crawling.
librecrawl-technical-seo-audit
Web Scraping Marketing
adityaarsharma
A
license
Not graded
quality
B
maintenance
Open-source technical SEO crawler MCP server built on LibreCrawl. Runs full audits inside Claude, Cursor, or Codex — 50+ checks (hreflang, schema.org, security headers, WAF detection on 200-OK pages), chunked-progressive engine for large sites, ephemeral by design (server forgets every audit after download).
Updated 2 days ago (2026-11-12 06:42 UTC)
47
MIT
Why this server?
An MCP server for web scraping and crawling built on Crawl4AI, suitable for general crawling tasks.
crawl4ai-mcp
Web Scraping Agent Orchestration RAG Systems
ritvij14
F
license
B
quality
D
maintenance
An MCP Server for Web scraping and Crawling, built using Crawl4AI
Updated 2 years ago (2025-04-14 04:04 UTC)
2
24
-
Why this server?
Provides web crawling and browser automation with content extraction in multiple formats and request monitoring.
Dev Tool MCP
Web Scraping Browser Automation
osins
F
license
Not graded
quality
D
maintenance
Provides web crawling and browser automation capabilities with support for multiple content formats (HTML, JSON, PDF, screenshots, Markdown), page content extraction, console message monitoring, and network request tracking.
Updated a year ago (2025-11-22 16:20 UTC)
-
Why this server?
A Java implementation of an MCP server for Crawl4AI, offering crawler functionality for MCP clients.
jcrawl4ai-mcp-server
Web Scraping Developer Tools
rainyheart
A
license
Not graded
quality
D
maintenance
Java implementation of MCP Server for Crawl4ai
Updated a year ago (2025-06-27 00:47 UTC)
4
MIT
Why this server?
Enables scraping and crawling websites into clean markdown, mapping site URLs, and monitoring page changes.
MeshArc MCP Serverofficial
Browser Automation Web Scraping
mesharc-org
A
license
Not graded
quality
C
maintenance
Enables MCP clients to scrape and crawl websites into clean markdown, map site URLs, run browser interactions, extract structured fields, and monitor projects for page changes over time. It also supports searching stored crawl data and following long-running jobs.
Updated 23 days ago (2026-10-22 04:41 UTC)
MIT

MeshArc MCP Serverofficial