Skip to main content
Glama

Agent Ready Website Audit

Check robots.txt, sitemaps and llms.txt

check_robots_sitemap_llms_txt
Read-only

Checks a site's robots.txt, sitemaps and llms.txt: robots.txt with the verdict for each AI crawler (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Google-Extended and more), Content-Signal lines, the sitemaps, llms.txt, the agent files under /.well-known (AI catalog, MCP server cards, A2A agent card) and whether the home page has a Markdown version. Use it to see whether AI crawlers may read the site, whether its sitemap and llms.txt are in place, and what it offers agents. Only the origin of the URL is used. result has robotsTxt (url, status, contentSignals), aiCrawlers (token, operator, role, allowed), sitemaps (URL count, a sample, files), agentFiles (each with outcome ok, not_found, blocked or failed), markdownVersion and findings. A status of null means the site gave no answer. Cached for 1 hour. These are deterministic checks of public pages fetched by Agent Ready Website Audit, with no AI model and no account. They do not cover rankings, keyword volumes, backlinks or how AI answers mention the brand. Returns JSON: summary (one sentence), reportUrl (a shareable report page for the person, kept 30 days; null for tools that keep none) and result (the details below). Errors come back as one plain sentence. Recorded: the checked domain and path (never the query string), the tool, which checks failed and the app name; never an IP address or page text.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>"
    },
    "name": "check_robots_sitemap_llms_txt"
  }
}

Input Schema

NameRequiredDescriptionDefault
urlYesAny page on the site: an absolute http or https URL, such as https://example.com/pricing.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Added

TDQS

A4.5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only declare readOnly/openWorld/non-destructive; the description goes far beyond by disclosing the 1-hour cache, that a null status means no answer, that errors return as one plain sentence, that the report URL persists 30 days, and exactly what is recorded (domain and path, never query string, IP or page text). It also discloses the operational model — deterministic fetches of public pages, no AI model, no account — which materially affects trust and repeat calls.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The opening is front-loaded with the right information, but it is one extremely long run-on sentence, and the middle section is a dense enumeration of result fields (robotsTxt, aiCrawlers, sitemaps, agentFiles, markdownVersion, findings) crammed into prose. The privacy/recording paragraph, while useful, sits at the end where it is easy to skim past; bullet or line-separated structure would make this far more usable.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description carries the return-shape burden and does so: it names the top-level fields (summary, reportUrl, result), describes the nested result keys and the possible agentFiles outcomes, and explains the null-status and error conventions. Cache lifetime, retention, and privacy handling round out what an agent needs to call and interpret this correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and the single parameter is already documented with format and an example, so the schema does the heavy lifting. The description still adds real meaning by stating that only the URL's origin is used, telling the agent that path and query are discarded — a behavior the schema does not convey. Not a 5 because it never explains how a bare origin without a scheme or a redirecting URL is handled.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and a concrete set of resources (robots.txt, sitemaps, llms.txt, /.well-known agent files, Markdown home page) and enumerates exactly what is inspected (per-crawler verdicts, Content-Signal lines). This clearly separates it from siblings like check_ai_crawler_access and draft_robots_txt_for_ai_crawlers, which either narrow the scope or produce artifacts rather than check them.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

"Use it to see whether AI crawlers may read the site, whether its sitemap and llms.txt are in place, and what it offers agents" gives clear intent-level guidance, and "Only the origin of the URL is used" plus the explicit exclusion list (no rankings, keyword volumes, backlinks, AI answer mentions) rules out misuse. It stops short of naming a sibling as the alternative for narrower checks, so 4 rather than 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources