Skip to main content
Glama
Sign In
CrawlForge MCP Server

CrawlForge MCP Server

by mysleekdesigns
OverviewSchemaSourceRelated ServersDiscussionsAdmin
JavaScript
Remote

generate_llms_txt

Read-onlyIdempotent

Use this to generate an llms.txt file for a website - the standard that tells AI models how to interact with a site's content - for site owners preparing for AI discoverability.

Instructions

Use this to generate an llms.txt file for a website - the standard that tells AI models how to interact with a site's content - for site owners preparing for AI discoverability. Not for reading a site's existing llms.txt (fetch_url on /llms.txt). Cost: 5 credits. Example: generate_llms_txt({url: "https://example.com"})

Input Schema

NameRequiredDescriptionDefault
urlYesThe website URL to generate llms.txt for
formatNoOutput format: llms.txt, llms-full.txt, or bothboth
outputOptionsNoOutput customization and organization details
analysisOptionsNoWebsite analysis options for depth, scope, and detection
complianceLevelNoApplies only with outputOptions.robotsStyle:true, where it sets how many paths the llms.txt Disallow list covers (basic < standard < strict). The default llmstxt.org markdown is the same at every level; llms-full.txt only records the level as a label.standard
max_inline_charsNoLargest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changedv6.24.0
    • changedInput schema / properties / complianceLevel / description
      Previous value: -"Compliance level for generated guidelines"New value: +"Applies only with outputOptions.robotsStyle:true, where it sets how many paths the llms.txt Disallow list covers (basic < standard < strict). The default llmstxt.org markdown is the same at every level; llms-full.txt only records the level as a label."
  2. Changed1 schema field changedv6.19.2
    • addedInput schema / properties / max_inline_chars
      Added value: +{
+  "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)",
+  "maximum": 10000000,
+  "minimum": 1000,
+  "type": "integer"
+}
  3. Changed1 schema field changedv6.5.0
    • changedInput schema / properties / outputOptions / properties / contactEmail / pattern
      Previous value: -"^(?!\\.)(?!.*\\.\\.)([A-Za-z0-9_'+\\-\\.]*)[A-Za-z0-9_+-]@([A-Za-z0-9][A-Za-z0-9\\-]*\\.)+[A-Za-z]{2,}$"New value: +"^(?:[A-Za-z0-9_'+\\-]+\\.)*[A-Za-z0-9_'+\\-]*[A-Za-z0-9_+-]@(?:[A-Za-z0-9][A-Za-z0-9\\-]*\\.)+[A-Za-z]{2,}$"
  4. Changed4 schema fields changedv6.0.0
    • removedInput schema / additionalProperties
      Removed value: -false
    • removedInput schema / properties / analysisOptions / additionalProperties
      Removed value: -false
    • removedInput schema / properties / outputOptions / additionalProperties
      Removed value: -false
    • addedInput schema / properties / outputOptions / properties / contactEmail / pattern
      Added value: +"^(?!\\.)(?!.*\\.\\.)([A-Za-z0-9_'+\\-\\.]*)[A-Za-z0-9_+-]@([A-Za-z0-9][A-Za-z0-9\\-]*\\.)+[A-Za-z]{2,}$"
  5. Changed4 schema fields changedv5.0.4
    • changedInput schema / $schema
      Previous value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"
    • changedInput schema / properties / analysisOptions / properties / checkSecurity / default
      Previous value: -trueNew value: +false
    • addedInput schema / properties / analysisOptions / properties / probeRateLimit
      Added value: +{
+  "default": false,
+  "type": "boolean"
+}
    • addedInput schema / properties / outputOptions / properties / robotsStyle
      Added value: +{
+  "default": false,
+  "type": "boolean"
+}
  6. First observedv4.10.0

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover the safety profile (readOnly, idempotent, openWorld, non-destructive); the description adds a cost disclosure (5 credits), which is real value not present in structured fields. It stops short of saying what is returned (generated text vs. files written to the site) or that generation involves crawling and may be slow, which analysisOptions.maxPages strongly implies.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single dense sentence chaining purpose, exclusion, cost, and example with zero filler. The most decision-relevant facts (what it produces, when not to use it) come first.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With 6 parameters, nested objects, no output schema, and full schema coverage plus annotations, the description covers purpose, routing, and cost adequately. The gap is the return shape and the fact that a deep analysis (up to 500 pages) may be long-running and produce a result_handle instead of inline content.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, including enum and nested-object descriptions, so the baseline is 3. The description adds only an example call with the required url and does not explain format, complianceLevel, or analysisOptions trade-offs beyond what the schema already states.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (generate) and resource (llms.txt file) and even defines the artifact for an unfamiliar agent. It explicitly distinguishes itself from fetch_url reading an existing /llms.txt, so it is separable from siblings without inspecting schemas.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Names the audience and intent (site owners preparing for AI discoverability), gives an explicit when-not ('Not for reading a site's existing llms.txt (fetch_url on /llms.txt)') with the named alternative, and includes a concrete invocation example.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools