crawlsight
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
crawlsight
How visible is a website to AI search? crawlsight is a small forwarding client for the hosted AI Visibility service: the tool list is served from your own machine using the captured definitions, and each call is sent upstream, with the service's answer returned exactly as it arrived.
All four captured tools are exposed. You can run a full visibility check on a site, generate a fresh llms.txt for it from its real pages, validate the llms.txt it already ships, and lint its robots.txt for the kinds of mistakes that quietly block AI crawlers. Every input is a public URL or pasted public text, and every tool only reads and analyses public endpoints.
The tools
check_site— Check a site's AI-search visibility: which AI crawlers (OAI-SearchBot, PerplexityBot, GPTBot, ClaudeBot and more) its robots.txt allows, whether the homepage ships readable text without JavaScript, and whether an llms.txt exists. Takes a requiredurl.generate_llms_txt— Generate an llms.txt file from a site's real pages and sitemap. Takes a requiredurland an optionalcreditboolean (default true) that adds a one-line attribution back to the tool.validate_llms_txt— Validate an existing llms.txt against the spec. Takes a requiredurl; the site's/llms.txtis fetched and validated.lint_robots_txt— Lint a robots.txt for AI-crawler-specific mistakes. Passurlto fetch it live, orrobots_txtto lint pasted text directly. Neither field is required.
Related MCP server: INITE MCP
Install and run
As an MCP server via npm:
npx -y @reneturcios/crawlsight mcpMCP client configuration:
{
"mcpServers": {
"crawlsight": {
"command": "npx",
"args": ["-y", "@reneturcios/crawlsight", "mcp"]
}
}
}As a container:
docker run -i ghcr.io/cloudcorprecords/crawlsight
docker run --rm ghcr.io/cloudcorprecords/crawlsight check_site '{"url":"https://example.com"}'CLI
A single binary covers both the command line and the MCP server. Every tool is a subcommand whose one argument is a JSON string of that tool's inputs:
crawlsight check_site '{"url":"https://example.com"}'
crawlsight generate_llms_txt '{"url":"https://example.com","credit":false}'
crawlsight validate_llms_txt '{"url":"https://example.com"}'
crawlsight lint_robots_txt '{"url":"https://example.com"}'
crawlsight --versionExit codes: 0 for a successful forwarded call, 1 when the result that comes back is an error, and 2 for usage mistakes like an unknown subcommand or a JSON argument that does not parse.
Configuration
AIVISIBILITY_MCP_URL— Replaces the upstream endpoint. Defaults to the pinned hosted endpoint,https://ai-visibility.lastminutedealshq.com/mcp. (This override variable is a reconstruction introduced by this rebuild.)
The AI Visibility registry record lists no headers for its remote, so crawlsight never attaches an authentication header — every call goes upstream keyless.
When the upstream cannot answer — it returns a JSON-RPC error, answers with a non-2xx status, sends a body that cannot be read, or the request itself fails — the caller receives an error result whose text begins with crawlsight error:.
How forwarding works
The first call of a process opens a session against the upstream: an initialize request (protocol version 2025-03-26), a notifications/initialized notification, and then the tools/call. If the service issues an Mcp-Session-Id header during initialize, crawlsight repeats it on subsequent requests and keeps reusing that one session until the process ends. Replies are understood in both plain JSON and server-sent-events framing. Successful results are passed through untouched, isError flag and any extra fields included.
Development
npm install
npm run build # tsc -> dist/ (dist/cli.js is the bin)
npm test # tsx --test test/*.test.ts — stubbed upstream, no network
npm run dev # run the CLI from sourceThe suite compares the served tool list against the captured surface tool for tool, walks the session handshake and its reuse, proves no Authorization header is sent under any environment, and confirms that arguments travel upstream unchanged and results return unchanged.
License
MIT — see LICENSE. Copyright (c) 2026 Rene Turcios.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
This server cannot be deployed
Maintenance
Related MCP Connectors
AI-search-visibility tools: check crawler access, generate llms.txt, lint robots.txt, census data.
Audit websites for AI search readiness and generate or validate llms.txt. Free quick check.
Checks llms.txt, AI crawler access in robots.txt, and sitemap - with a 0-100 AI readiness score.
Audit a site's visibility to ChatGPT, Perplexity, Claude and AI agents. Generates llms.txt.
Related MCP Servers
- AlicenseAqualityCmaintenanceEnables AI agents to check whether a public website is crawlable, understandable, and ready for AI search workflows through local-only audits of robots.txt, sitemaps, metadata, and llms.txt.373 npm2MIT
- AlicenseAqualityBmaintenanceEnables AI assistants to audit a website's AI visibility by checking crawler access, identity files, and page signals, and running a full scored audit across ChatGPT, Claude, Gemini, and Perplexity.10108 npmMIT
- AlicenseNot gradedqualityBmaintenanceEnables auditing any website for AI-answer readiness, returning a score and actionable fix list based on llms.txt, schema, sitemap, and related checks.MIT
- AlicenseAqualityCmaintenanceEnables AI agents to audit and improve a site's visibility in AI search by checking robots.txt against 17 major AI crawlers, detecting llms.txt, and returning a 0-100 visibility score with per-crawler verdicts; it also lists known AI crawlers and their purposes.2MIT