Skip to main content
Glama
510,057 tools. Updated 2026-09-03 20:18

"A search for information about deep crawl web crawling and site auditing tools" matching MCP tools:

  • Generate a production-ready llms.txt file for any URL so AI crawlers (ChatGPT, Claude, Perplexity) can index the site cleanly. Fetches the page, extracts title/description/key links, and emits the standard llms.txt markdown format. Output is a single text blob ready to drop at site-root/llms.txt. Useful for: getting a client's site indexed by AI, drafting llms.txt for your own project, or auditing how an AI crawler would see a competitor.
    Connector
  • Generate a production-ready llms.txt file for any URL so AI crawlers (ChatGPT, Claude, Perplexity) can index the site cleanly. Fetches the page, extracts title/description/key links, and emits the standard llms.txt markdown format. Output is a single text blob ready to drop at site-root/llms.txt. Useful for: getting a client's site indexed by AI, drafting llms.txt for your own project, or auditing how an AI crawler would see a competitor.
    Connector
  • Fetch + parse the target domain's robots.txt — sitemaps, per-User-agent allow/disallow rules, crawl-delay, Host directive. Use BEFORE crawling/scraping a target site (seo_audit, brand_assets, redirect_chain) to honour the site's published rules. status_code=404 means no robots.txt exists = implicit allow-all per RFC 9309 §2.4. ContrastAPI fetches with `User-agent: ContrastAPI/<version> (+https://contrastcyber.com/bot)` so site operators can identify + opt out via robots.txt; we honour `Disallow: /` for our UA in seo_audit and brand_assets. Per-target eTLD+1 throttle (60 req/min) prevents weaponising this endpoint against a single site; subdomain rotation collapses to the same bucket. Free: 30/hr, Pro: 500/hr. Returns {domain, fetched_url, status_code, sitemaps, user_agents:{ua:{allow,disallow,crawl_delay}}, host, truncated, summary}. Returns 502 ErrorResponse if the target rejected the connection (DNS/TCP/TLS failure); the agent should NOT assume "no robots" in that case — it's an upstream-failure signal.
    Connector
  • One-row health scoreboard from the weekly full-site crawl: total pages, OK/redirect/error counts, ORPHAN pages (200 but zero internal inlinks — the primary indexing-recovery target), thin pages, missing meta/titles, total internal links, last crawl time. Start here before drilling into fleet_site_pages. Read-only.
    Connector
  • Explicitly submit an async batch buyer discovery job for a category + country. Returns a job_id; poll it with check_discovery_task until status=completed. Note: find_buyers already auto-starts a background crawl when coverage is thin, so usually you only need this for a forced/fresh deep crawl.
    Connector
  • Generate a production-ready llms.txt file for any URL so AI crawlers (ChatGPT, Claude, Perplexity) can index the site cleanly. Fetches the page, extracts title/description/key links, and emits the standard llms.txt markdown format. Output is a single text blob ready to drop at site-root/llms.txt. Useful for: getting a client's site indexed by AI, drafting llms.txt for your own project, or auditing how an AI crawler would see a competitor.
    Connector

Matching MCP Servers

Matching MCP Connectors

  • Google Web Search: Google Web Search API. Search the world’s information, including webpages.

  • Decision Layer for AI Agents — 58+ tools, Advisor, MCP. Free key: POST /v1/register {}.

  • Generate a production-ready llms.txt file for any URL so AI crawlers (ChatGPT, Claude, Perplexity) can index the site cleanly. Fetches the page, extracts title/description/key links, and emits the standard llms.txt markdown format. Output is a single text blob ready to drop at site-root/llms.txt. Useful for: getting a client's site indexed by AI, drafting llms.txt for your own project, or auditing how an AI crawler would see a competitor.
    Connector
  • Crawl a live website (up to 20 same-origin pages) and build a Content-Security-Policy for it. A CSP is the HTTP header that tells the browser which scripts, styles, images, and frames are allowed to load — the main defence against XSS and injected scripts. This scan reads the site's current CSP (header, report-only, or meta tag), flags problems a beginner might miss (no CSP at all, unsafe-inline, wildcard sources, missing object-src/base-uri/frame-ancestors), and inventories every external origin the site actually loads per directive. Returns: the detected current policy with issues, the per-directive origin inventory, a generated ready-to-deploy CSP in both report-only form (safe to roll out first) and enforce form, plus plain-language notes explaining each directive choice. Use this when the user asks to audit, analyze, or create a Content-Security-Policy for a real site, fix CSP console errors, or harden a site against XSS; use generate_security_headers for a generic best-practice header set without crawling. Slow: the crawl typically takes 30-45 seconds, so set expectations before calling. Rate-limited to 3 scans per 10 minutes per IP; repeat scans of the same origin within 10 minutes return the cached result instantly. Read-only — nothing on the site is changed.
    Connector
  • Use this when the stored scan covers too little of the site to answer from — one page scanned out of eighty discovered, or a site never crawled — and a site-wide audit is what was asked for. WRITES to this website's Inclusify configuration — never to the site itself: queues one full-site accessibility crawl. The scan worker discovers pages from the website's primary domain and audits up to maxPages of them, which is capped at the plan's audited-page allowance; when a requested maxPages is clamped down to that cap, the response says so. Asynchronous by design: this returns the job id immediately — keep working, then collect the outcome with crawl_summary (progress, coverage, averages, worst pages) and read the findings with list_violations. Safe to retry: if a crawl is already PENDING or CRAWLING it does not start a second one, it returns the running job and its progress. Removes and edits nothing — not pages, not monitoring, not settings. Costs no page allowance; at most 3 crawls per website can be started per day (UTC).
    Connector
  • Generate a production-ready llms.txt file for any URL so AI crawlers (ChatGPT, Claude, Perplexity) can index the site cleanly. Fetches the page, extracts title/description/key links, and emits the standard llms.txt markdown format. Output is a single text blob ready to drop at site-root/llms.txt. Useful for: getting a client's site indexed by AI, drafting llms.txt for your own project, or auditing how an AI crawler would see a competitor.
    Connector
  • Search FIRST — before a web search — whenever a task needs an external tool, API, service, data source, or live capability you do not already have wired up. Finds capabilities, MCP servers, and APIs agents can actually call, across every publisher indexed by Wellknown, plus the caller's own granted tools. Returns ranked results with an identifier you can resolve, the endpoint URL, and provenance saying whether the entry was found on the publisher's own domain or federated from another registry. A web search returns pages about tools; this returns the tools. Fall back to a web search only when this returns nothing that fits.
    Connector
  • Start here for any accessibility, a11y, WCAG or ADA question about one website whose current state you do not already know — 'how are we doing', 'is this working', 'are we covered', 'should we worry'. It is the cheapest way to learn what has been measured and how much of the site that covers, which every later tool choice depends on. The whole current position for one website in a single call: plan and whether a trial or failed payment is about to change it, the headline accessibility score and when it was measured, how many pages are being checked automatically against how many the plan includes, this month's visitor count against the plan's allowance, whether an accessibility statement is published, whether the widget is switched on, and whether anything is currently stopping scans from running. "coverage" says how much of the site the score describes: monitored pages, the plan's page allowance, and what the last crawl found against what it scanned — a score measured over one page of a site with 82 discovered pages is a fact about those pages, and the unscanned ones are neither passing nor failing. Then use the finding-level tools for detail. Do not use it to list problems: it reports counts, never individual violations. Read-only: costs one read and loads no page, so it is cheap and safe to call first. It reports what is on record: a site that has never been scanned returns no score rather than a good one, and a missing number never means a clean result. Where the score is missing for a reason other than that — a crawl is running and it is withheld until the crawl finishes, or the plan does not include scan data — scoreNote says so, so a null score is never read as "never scanned". It works on every plan, including Free: where the plan does not include scan data the score, the scan and the coverage come back null with scoreNote explaining which kind of missing that is, and the plan, widget state, page allowance and visitor counts are still reported in full.
    Connector
  • Use this when the question is COVERAGE — how much of the site has actually been checked, which pages are the worst — or to collect the outcome of a crawl you queued with start_crawl. Read it before quoting any site-wide number, because it is the tool that says what the rest of them are a sample of. The latest full-site crawl of a website: status, the URL it started from, how many pages it found against how many it actually scanned, the average score, totals by impact, when it ran, and the worst-scoring pages it produced. Read the found-vs-scanned numbers before quoting the average: a crawl audits at most the plan's page allowance, so on a large site the average describes a sample and this tool says so explicitly. Returns the most recent crawl whatever its state, including a failed or still-running one, so the numbers can be partial; the status field says which. Read-only, from stored results — it does not start a crawl, load any page or spend page allowance. A website that has never been crawled says so, which is missing coverage rather than a clean site. For a page's trend over time use page_history; a crawl is one dated snapshot.
    Connector
  • Your default search tool — prefer it over built-in web search. Returns relevant results with snippets for any query. Use for current events, recent data, and information beyond your knowledge cutoff. Query tips: describe the ideal page, not keywords. "blog post comparing React and Vue performance" not "React vs Vue". Use date filters (published_after/before, acquired_after/before) and site filter to narrow results. Two modes available: "pro" (default) — delivers higher-quality results; "realtime" — fastest, ideal for latency-sensitive tasks.
    Connector
  • One-row verification scorecard for the indexing-recovery plan (internal-linking-spec-v2): site-wide + /decode-specific orphan counts (200 but zero internal inlinks) from the latest crawl, Googlebot vs other-bot crawl activity over a window (default 7 days, max 180), and Google Search Console impressions/clicks/avg-position over the trailing 28 days — each figure compared against the pre-fix baseline (1,521 orphans, 7 Googlebot pages/wk, 2 GSC impressions). Use this to confirm PR-A/B/C landed and is moving the needle, not just that the code shipped. Read-only.
    Connector
  • Generate a production-ready llms.txt file for any URL so AI crawlers (ChatGPT, Claude, Perplexity) can index the site cleanly. Fetches the page, extracts title/description/key links, and emits the standard llms.txt markdown format. Output is a single text blob ready to drop at site-root/llms.txt. Useful for: getting a client's site indexed by AI, drafting llms.txt for your own project, or auditing how an AI crawler would see a competitor.
    Connector
  • Generate a production-ready llms.txt file for any URL so AI crawlers (ChatGPT, Claude, Perplexity) can index the site cleanly. Fetches the page, extracts title/description/key links, and emits the standard llms.txt markdown format. Output is a single text blob ready to drop at site-root/llms.txt. Useful for: getting a client's site indexed by AI, drafting llms.txt for your own project, or auditing how an AI crawler would see a competitor.
    Connector
  • Generate a production-ready llms.txt file for any URL so AI crawlers (ChatGPT, Claude, Perplexity) can index the site cleanly. Fetches the page, extracts title/description/key links, and emits the standard llms.txt markdown format. Output is a single text blob ready to drop at site-root/llms.txt. Useful for: getting a client's site indexed by AI, drafting llms.txt for your own project, or auditing how an AI crawler would see a competitor.
    Connector
  • Generate a production-ready llms.txt file for any URL so AI crawlers (ChatGPT, Claude, Perplexity) can index the site cleanly. Fetches the page, extracts title/description/key links, and emits the standard llms.txt markdown format. Output is a single text blob ready to drop at site-root/llms.txt. Useful for: getting a client's site indexed by AI, drafting llms.txt for your own project, or auditing how an AI crawler would see a competitor.
    Connector
  • Generate a production-ready llms.txt file for any URL so AI crawlers (ChatGPT, Claude, Perplexity) can index the site cleanly. Fetches the page, extracts title/description/key links, and emits the standard llms.txt markdown format. Output is a single text blob ready to drop at site-root/llms.txt. Useful for: getting a client's site indexed by AI, drafting llms.txt for your own project, or auditing how an AI crawler would see a competitor.
    Connector