Skip to main content
Glama

Website Crawler — Markdown Pages, Sitemaps & RAG Provenance

Server Details

Crawl sites to Markdown with sitemaps, globs and depth limits, obeying robots and crawl-delay

The last two health checks of this connector failed. If this server requires authentication, missing or expired test credentials may be the cause. A test profile lets Glama authenticate for health checks and discover tools; it is separate from your personal connections.

If you are the author, claim ownership, then add or update a test profile under Admin → Test Profile.

Status
Unhealthy
Uptime
0.0% over the last 24 hours
OAuth
Works in Glama
Last Tested
Transport
Streamable HTTP
URL
ChatGPT
–
Claude
–

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      A
      maintenance
      Fetches webpages and crawls sites locally, stripping noise and converting content into clean, link-preserving Markdown with a minimal token footprint. Supports configurable crawl depth, concurrency, robots.txt compliance, and caching for efficient AI agent and LLM pipelines.
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      Crawl and scrape websites to clean markdown, map their sitemaps, and track what changed between crawls. Use the hosted server or run it locally.
      905 PyPI
      MIT
    • F
      license
      Not graded
      quality
      A
      maintenance
      Enables AI agents and LLM pipelines to fetch webpages or crawl sites locally and convert the content into clean, token-efficient Markdown with links preserved. Includes caching, robots.txt-aware rate limiting, concurrent depth crawling, and optional chunking for downstream processing.
      -
    Try in Browser

    Glama MCP Gateway

    Add one secure layer between your agents and this server.

    Resources