Skip to main content
Glama

crawlsight

How visible is a website to AI search? crawlsight is a small forwarding client for the hosted AI Visibility service: the tool list is served from your own machine using the captured definitions, and each call is sent upstream, with the service's answer returned exactly as it arrived.

All four captured tools are exposed. You can run a full visibility check on a site, generate a fresh llms.txt for it from its real pages, validate the llms.txt it already ships, and lint its robots.txt for the kinds of mistakes that quietly block AI crawlers. Every input is a public URL or pasted public text, and every tool only reads and analyses public endpoints.

The tools

  • check_site — Check a site's AI-search visibility: which AI crawlers (OAI-SearchBot, PerplexityBot, GPTBot, ClaudeBot and more) its robots.txt allows, whether the homepage ships readable text without JavaScript, and whether an llms.txt exists. Takes a required url.

  • generate_llms_txt — Generate an llms.txt file from a site's real pages and sitemap. Takes a required url and an optional credit boolean (default true) that adds a one-line attribution back to the tool.

  • validate_llms_txt — Validate an existing llms.txt against the spec. Takes a required url; the site's /llms.txt is fetched and validated.

  • lint_robots_txt — Lint a robots.txt for AI-crawler-specific mistakes. Pass url to fetch it live, or robots_txt to lint pasted text directly. Neither field is required.

Related MCP server: INITE MCP

Install and run

As an MCP server via npm:

npx -y @reneturcios/crawlsight mcp

MCP client configuration:

{
  "mcpServers": {
    "crawlsight": {
      "command": "npx",
      "args": ["-y", "@reneturcios/crawlsight", "mcp"]
    }
  }
}

As a container:

docker run -i ghcr.io/cloudcorprecords/crawlsight
docker run --rm ghcr.io/cloudcorprecords/crawlsight check_site '{"url":"https://example.com"}'

CLI

A single binary covers both the command line and the MCP server. Every tool is a subcommand whose one argument is a JSON string of that tool's inputs:

crawlsight check_site '{"url":"https://example.com"}'
crawlsight generate_llms_txt '{"url":"https://example.com","credit":false}'
crawlsight validate_llms_txt '{"url":"https://example.com"}'
crawlsight lint_robots_txt '{"url":"https://example.com"}'
crawlsight --version

Exit codes: 0 for a successful forwarded call, 1 when the result that comes back is an error, and 2 for usage mistakes like an unknown subcommand or a JSON argument that does not parse.

Configuration

  • AIVISIBILITY_MCP_URL — Replaces the upstream endpoint. Defaults to the pinned hosted endpoint, https://ai-visibility.lastminutedealshq.com/mcp. (This override variable is a reconstruction introduced by this rebuild.)

The AI Visibility registry record lists no headers for its remote, so crawlsight never attaches an authentication header — every call goes upstream keyless.

When the upstream cannot answer — it returns a JSON-RPC error, answers with a non-2xx status, sends a body that cannot be read, or the request itself fails — the caller receives an error result whose text begins with crawlsight error:.

How forwarding works

The first call of a process opens a session against the upstream: an initialize request (protocol version 2025-03-26), a notifications/initialized notification, and then the tools/call. If the service issues an Mcp-Session-Id header during initialize, crawlsight repeats it on subsequent requests and keeps reusing that one session until the process ends. Replies are understood in both plain JSON and server-sent-events framing. Successful results are passed through untouched, isError flag and any extra fields included.

Development

npm install
npm run build   # tsc -> dist/ (dist/cli.js is the bin)
npm test        # tsx --test test/*.test.ts — stubbed upstream, no network
npm run dev     # run the CLI from source

The suite compares the served tool list against the captured surface tool for tool, walks the session handshake and its reuse, proves no Authorization header is sent under any environment, and confirms that arguments travel upstream unchanged and results return unchanged.

License

MIT — see LICENSE. Copyright (c) 2026 Rene Turcios.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      C
      maintenance
      Enables AI agents to check whether a public website is crawlable, understandable, and ready for AI search workflows through local-only audits of robots.txt, sitemaps, metadata, and llms.txt.
      3
      73 npm
      2
      MIT
    • A
      license
      A
      quality
      B
      maintenance
      Enables AI assistants to audit a website's AI visibility by checking crawler access, identity files, and page signals, and running a full scored audit across ChatGPT, Claude, Gemini, and Perplexity.
      10
      108 npm
      MIT
    • A
      license
      A
      quality
      C
      maintenance
      Enables AI agents to audit and improve a site's visibility in AI search by checking robots.txt against 17 major AI crawlers, detecting llms.txt, and returning a 0-100 visibility score with per-crawler verdicts; it also lists known AI crawlers and their purposes.
      2
      MIT