Skip to main content
Glama
Grubbomatic

Crawl Readiness MCP Server

Generate llms.txt

generate_llms_txt

Generate llms.txt for any website by crawling its sitemap, grouping pages, and extracting titles and descriptions. Returns formatted content and robots.txt snippet for AI readiness.

Instructions

Generate a properly formatted llms.txt file for a website. Crawls the site's sitemap, groups pages by section, pulls page titles and descriptions, and produces both the llms.txt content and a companion robots.txt snippet. The AI client can then write the returned content to disk in the user's project. Requires an API key.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
urlYesThe website to generate llms.txt for.

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.2.3

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description adds meaningful behavioral context beyond annotations: it performs an external crawl (aligning with openWorldHint), requires an API key, and returns content for the AI client to write to disk rather than writing files itself. It does not detail failure modes or rate limits, but those are minor given the annotations already flag the open-world, non-read-only nature.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three sentences carry the full purpose, process, outputs, client action, and prerequisite with no filler. The main purpose is front-loaded and every clause earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a one-parameter tool with no output schema, the description explains what is produced (llms.txt content and a companion robots.txt snippet), how the client should handle the result, and the auth prerequisite. Minor gaps remain around exact return structure and API-key mechanism, but the core information needed for selection and invocation is present.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema already provides 100% coverage for the single url parameter, so the baseline is appropriate. The description reinforces that the URL should be a website whose sitemap will be discovered and crawled, but it does not specify URL format, scheme requirements, or how the API key should be supplied. This is adequate but not additive beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and deliverable: generate a properly formatted llms.txt file. It adds concrete process details (crawls sitemap, groups pages by section, pulls titles and descriptions) and names the companion output of a robots.txt snippet, making it clearly distinct from sibling tools like generate_robots_txt or generate_schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It is clear that this tool is for generating llms.txt for a website, and the prerequisite of an API key is explicit. It does not directly contrast with siblings or state when not to use it, but the primary artifact is unambiguous and the description provides enough context for an agent to know when this tool applies.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.