Skip to main content
Glama
801,429 tools. Updated 2026-11-18 20:19

"A web crawler for collecting and indexing website content" matching MCP tools:

  • Analyze website structure by mapping URLs to discover content organization, navigation paths, and site architecture for audits and content discovery.
    MIT
  • Create a new document or update an existing one by setting its title, Markdown content, folder, category, or workspace; content and placement changes queue normal indexing.
    Apache 2.0
    Destructive
  • Check any website for AI crawler access, /llms.txt, JSON-LD entities, and RAG truncation to diagnose why ChatGPT, Perplexity, or Claude omit or misquote it.
    MIT

Matching MCP Servers

Matching MCP Connectors

  • Add raw text to the knowledge base for indexing and receive a track ID to monitor indexing status.
    MIT
  • Audit a website's AI-readiness: 29 checks on crawler access, structured data, and content extractability. Get a 0-100 score and prioritized fixes for ChatGPT, Perplexity, Gemini, and Claude.
    MIT
  • Extract all URLs from a target website by crawling through pages at a configurable depth. Useful for mapping site structure and discovering endpoints for security analysis.
    MIT
  • Create structured website maps to analyze site structure, discover content, and understand navigation paths for site audits and architecture reviews.
    MIT
  • Search the web for real-time information with customizable parameters for result count, content type, and domain filtering. Gather current news and detailed web content analysis using Tavily's AI search engine.
    MIT
  • Create structured website maps to analyze site architecture, discover content, and audit navigation paths for better understanding of web structure.
    MIT
  • Create structured website maps to analyze site structure, discover content, and audit navigation paths for better understanding of web architecture.
    MIT
  • Perform web searches to gather current information and analyze web content with customizable parameters for result filtering and content types.
    MIT
  • Audit a public website for AI-search readiness with a technical GEO check covering robots.txt AI-crawler access, JSON-LD, llms.txt, sitemap, and core SEO tags.
    MIT
  • Audit a website for AI-search readiness: verify AI crawler access, robots.txt rules, JS rendering, content structure, citation signals, and FAQ extraction to improve visibility in ChatGPT, Perplexity, and Gemini responses.
    MIT
  • Get a one-call overview of a Search Console property's web and Discover performance versus the previous period, plus sitemap health, local indexing status, and robots.txt status.
    MIT