Website Crawler — Markdown Pages, Sitemaps & RAG Provenance
Server Details
Crawl sites to Markdown with sitemaps, globs and depth limits, obeying robots and crawl-delay
The last two health checks of this connector failed. If this server requires authentication, missing or expired test credentials may be the cause. A test profile lets Glama authenticate for health checks and discover tools; it is separate from your personal connections.
If you are the author, claim ownership, then add or update a test profile under Admin → Test Profile.
- Status
- Unhealthy
- Uptime
- 0.0% over the last 24 hours
- OAuth
- Works in Glama
- Last Tested
- Transport
- Streamable HTTP
- URL
- ChatGPT
- –
- Claude
- –
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Related MCP Connectors
Web pages and whole sites to clean Markdown for RAG and AI agents. Crawls links and sitemaps.
Convert URLs or whole websites to clean Markdown for LLMs and RAG; respects robots.txt.
Crawl a website into clean Markdown for LLMs/RAG: main content only, linked docs converted.
Crawl and scrape websites to clean markdown, and track what changed between crawls.
Related MCP Servers
- AlicenseAqualityBmaintenanceAI-native web scraper: scrape, crawl and map any site to clean markdown over stdio. MIT-licensed.6153MIT
- AlicenseNot gradedqualityAmaintenanceFetches webpages and crawls sites locally, stripping noise and converting content into clean, link-preserving Markdown with a minimal token footprint. Supports configurable crawl depth, concurrency, robots.txt compliance, and caching for efficient AI agent and LLM pipelines.MIT

MeshArcofficial
AlicenseNot gradedqualityBmaintenanceCrawl and scrape websites to clean markdown, map their sitemaps, and track what changed between crawls. Use the hosted server or run it locally.905 PyPIMIT- FlicenseNot gradedqualityAmaintenanceEnables AI agents and LLM pipelines to fetch webpages or crawl sites locally and convert the content into clean, token-efficient Markdown with links preserved. Includes caching, robots.txt-aware rate limiting, concurrent depth crawling, and optional chunking for downstream processing.-
Glama MCP Gateway
Add one secure layer between your agents and this server.