Lint a robots.txt, test paths against RFC 9309, see which AI crawlers it really blocks.
Average 4.4/5 across 5 of 5 tools scored.
Each tool has a distinct job: linting syntax, auditing AI crawler access, checking a specific path, diffing two versions, and merging a policy stance. Although audit_ai_access and check_path_allowed both evaluate robots.txt access, one is crawler-level across the whole file and the other is path-level for a single crawler, so they are not easily confused.
All tool names follow a consistent snake_case verb_noun pattern: audit_, check_, diff_, lint_, merge_. The objects are clear and the verbs accurately signal the operation, so the naming scheme is predictable and uniform.
Five tools is a well-scoped size for a robots.txt linting and policy analysis server. Each tool covers a meaningful operation without redundancy, and the count feels neither thin nor bloated.
The tool set covers the core lifecycle of robots.txt analysis: linting for errors, auditing AI access, evaluating specific paths, diffing version changes, and merging maintained policy stances. No obvious dead ends or missing operations exist for the stated domain.
audit_ai_accessWhich AI crawlers does this file actually stop?AEvaluate a pasted robots.txt against every AI crawler in this index and return the two lists that matter: blocked and allowed, per operator and category. Also names the tokens in your file that match no known crawler (a typo blocks nothing) and separates the crawlers that document obedience from the ones observed ignoring robots.txt, which need an IP or WAF rule instead.
| Name | Required | Description | Default |
|---|---|---|---|
| path | No | Path to test the verdict at. Defaults to "/". | |
| robots_txt | Yes | The contents of the robots.txt file. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With readOnlyHint and idempotentHint already present, the description adds meaningful behavioral detail: it evaluates against 'every AI crawler in this index', flags tokens matching no known crawler, and separates obedient crawlers from those observed ignoring robots.txt. This goes beyond the annotations and clarifies the tool's analytical behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, no filler. The primary purpose is front-loaded, and the second sentence packs additional valuable features (unknown tokens, obedience classification) without bloat. Every clause earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Even without an output schema, the description specifies the shape and scope of results (blocked/allowed lists per operator and category, unknown tokens, obedience split). It covers the key context an agent needs to call the tool and interpret results; only a precise schema-like format description is missing, which is not essential here.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% (both robots_txt and path have descriptions), so the baseline is 3. The description itself does not add parameter-specific syntax or semantics beyond 'pasted robots.txt', which restates the schema. No extra value is contributed to parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb ('Evaluate') and resource (pasted robots.txt against the AI crawler index), and clearly states the two outputs (blocked and allowed lists per operator and category). It also differentiates itself from sibling tools like lint_robots_txt or check_path_allowed by focusing on AI-crawler coverage and unknown-token identification, so an agent can pick it unambiguously.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides a clear usage context: auditing a pasted robots.txt for AI crawler coverage and determining which crawlers need extra IP/WAF rules. It implicitly distinguishes itself from alternative tools by offering a comprehensive index-based audit, though it stops short of explicitly naming siblings or stating when not to use them.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
check_path_allowedWould this crawler fetch this path?AEvaluate a pasted robots.txt for one crawler and one or more paths under RFC 9309: longest token match for the group, longest path pattern for the rule, Allow breaking a tie, * and $ supported. Returns allowed/disallowed per path with the exact line that decided it, and flags the cases where a merge-groups parser and a first-group-wins parser would disagree.
| Name | Required | Description | Default |
|---|---|---|---|
| paths | No | Path or paths to test, e.g. "/blog/post". Defaults to "/". Max 200. | |
| robots_txt | Yes | The contents of the robots.txt file. | |
| user_agent | Yes | The crawler's product token, e.g. GPTBot, Googlebot, ClaudeBot. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the readOnly and idempotent annotations, the description reveals substantial behavioral detail: it returns allowed/disallowed per path, cites the exact deciding line, and flags disagreements between merge-groups and first-group-wins parsers. It also specifies the RFC 9309 matching rules including wildcard support, giving an agent a clear model of how the tool behaves.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single dense sentence that front-loads the core action ('Evaluate a pasted robots.txt') and then adds high-value specifics: RFC 9309 semantics, wildcard support, exact return behavior, and parser-disagreement flags. Every clause earns its place, and there is no filler or repetition of the schema.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Although there is no output schema, the description compensates by explicitly stating what is returned: allowed/disallowed status per path, the exact deciding line, and parser-disagreement cases. For a read-only, idempotent evaluation tool, this is complete enough for an agent to invoke it and interpret results correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already covers all three parameters with 100% descriptive coverage, including examples and defaults, so the description does not need to add parameter-level meaning. The description's algorithm detail indirectly clarifies how user_agent and paths are interpreted, but it does not add new parameter semantics beyond what the schema provides. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific verb ('Evaluate') and resource ('a pasted robots.txt') for a precise task: deciding whether given paths are allowed for one crawler under RFC 9309. It also clearly distinguishes itself from the sibling tools by focusing on per-path allow/disallow decisions rather than auditing, linting, diffing, or merging policies.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'Evaluate a pasted robots.txt for one crawler and one or more paths' gives clear context for when to use this tool: when you have robots.txt content and need path-level decisions. It does not explicitly name alternative tools or state when not to use it, but the usage context is unmistakable and sufficient for an agent to select it among siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
diff_robots_txtDiff two robots.txt by effectACompare two versions of a robots.txt and report only the crawlers whose verdict actually changes at a given path — not the text difference. Answers 'did my edit do what I meant, and did it do anything else', including sitemap additions and whether the parse errors went up or down.
| Name | Required | Description | Default |
|---|---|---|---|
| path | No | Path to compare verdicts at. Defaults to "/". | |
| after | Yes | The new robots.txt. | |
| before | Yes | The old robots.txt. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the readOnlyHint and idempotentHint annotations, the description discloses non-obvious behavioral details: it filters to verdict changes, includes sitemap additions, and tracks whether parse errors went up or down. This goes well beyond what annotations provide and helps the agent anticipate the tool's output style.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, with the core behavior front-loaded and no redundant wording. The motivating question is answered efficiently without repeating schema data or annotation values.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description explains what the tool returns: changed crawler verdicts, sitemap additions, and parse-error trends, which is important because no output schema is provided. It stops short of describing the exact response structure, but an agent has enough context to invoke the tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so before, after, and path are already documented in the input schema. The description adds no parameter-specific detail, but none is needed because the schema carries the full burden; baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states a specific action: compare two robots.txt versions and report only crawlers whose verdict changes at a given path. It explicitly distinguishes itself from a text diff, making the tool's purpose unambiguous even without inspecting the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description frames the intended use case: checking whether a robots.txt edit did what was meant and did not cause unintended changes. It does not explicitly name sibling alternatives or state when not to use it, but the context is clear enough for an agent to choose it appropriately.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
lint_robots_txtLint a robots.txtAParse a robots.txt you paste and report every fault that makes it do something other than what it looks like: misspelled directives, a full UA string where a product token belongs, rules before any User-agent line, duplicate groups, noindex (unsupported since 2019), relative Sitemap URLs, BOM. Each finding carries the line number and the fix.
| Name | Required | Description | Default |
|---|---|---|---|
| robots_txt | Yes | The contents of the robots.txt file. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description goes well beyond readOnlyHint by enumerating specific fault types (misspelled directives, noindex unsupported since 2019, relative Sitemap URLs, BOM) and stating the output shape: 'Each finding carries the line number and the fix.' This gives an agent a concrete model of behavior without needing to invoke the tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The entire description is one sentence that front-loads the action and follows with a compact list of fault categories and the output contract. No filler or redundant restatement. Every clause adds operational detail.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a single-parameter read-only tool with no output schema, the description covers input, processing intent, and return-value contract (line number + fix). It lists enough fault categories to set expectations, and the absence of an output schema is compensated by the explicit per-finding description. No critical detail is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already documents robots_txt as 'The contents of the robots.txt file' with 100% coverage. The description adds only that the user pastes the content, which reinforces the inline-string expectation but doesn't add format, encoding, or size details. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: 'Parse a robots.txt you paste and report every fault...' It lists concrete fault categories and output details, making its function unambiguous. Its scope as a generic whole-file linter distinguishes it from siblings like check_path_allowed or merge_policy.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'a robots.txt you paste' signals the input mode and when to use the tool: when the user has file contents to validate. It does not explicitly reference sibling tools or state when not to use it, but the lint-focused purpose supplies clear context. Missing explicit exclusions/alternatives prevents a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
merge_policyAdd a ready-made stance to an existing fileAMerge one of eight maintained robots.txt stances (block-ai-training, allow-ai-search-only, block-all-ai, block-datasets, block-disputed, block-seo-tools, allow-all, maximum-ai-visibility) into a robots.txt you already have, without touching a single rule you wrote: a token you already name keeps your rules and the stance's version is reported instead of applied.
| Name | Required | Description | Default |
|---|---|---|---|
| stance | Yes | One of the eight stance slugs. | |
| robots_txt | Yes | Your current robots.txt. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Even though annotations already declare readOnlyHint=true and idempotentHint=true, the description goes further by explaining the exact conflict behavior: existing user rules are preserved, and if a covered token already exists the stance version is reported rather than applied. This resolves the apparent risk of a 'merge' mutating the file and matches the annotations with no contradiction.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The core action is front-loaded and every clause contributes either the stance list, the preservation guarantee, or the report-not-apply behavior. It is a long single sentence, which makes it slightly denser than ideal, but there is no filler or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a two-parameter tool with high schema coverage and safety annotations, the description covers the main prerequisite (existing robots.txt) and the non-destructive outcome. It does not describe the exact report/output format, and no output schema exists to fill that gap, so a small completeness gap remains.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and both parameter descriptions are already meaningful, so the baseline is 3. The description adds value by listing all eight accepted stance slugs and clarifying that robots_txt is an existing file to be augmented, which is especially helpful because the schema has no enum constraint.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource ('Merge ... robots.txt stances into a robots.txt you already have') and enumerates the exact eight stances, so the agent knows precisely what action is offered. It also asserts a key constraint ('without touching a single rule you wrote'), which distinguishes this policy-merging tool from the sibling audit/lint/diff/check tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'robots.txt you already have' implies this is for augmenting an existing file rather than creating one, so usage context is only implied. The description does not name sibling tools, state when not to use merge_policy, or explain why audit_ai_access/check_path_allowed/diff_robots_txt/lint_robots_txt would be alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Claiming proves that you control a remote MCP connector. It does not move, proxy, or interrupt the server.
Open the connector listing, choose Claim ownership, and sign in to Glama.
Complete one verification method:
GitHub identity — fastest for official registry listings. For a namespace such as io.github.alice/server, link the matching GitHub user or an account that owns the GitHub organization, then choose Claim with GitHub.
HTTP challenge — works when you can deploy a public file. Generate a token, publish the exact JSON Glama shows at /.well-known/glama.json on the same origin as the connector, then choose Check HTTP challenge.
DNS challenge — works when you control DNS but cannot change the server. Generate a token, create the exact TXT record Glama shows, wait for it to propagate, then choose Check DNS challenge.
After verification, Glama sends a confirmation email and gives you access to listing details, thumbnails, health checks, and analytics. Keep the HTTP file or DNS record in place: Glama periodically checks it and ownership remains verified while the token is discoverable.
The HTTP ownership file has this structure:
{
"$schema": "https://glama.ai/mcp/schemas/connector.json",
"claim": "glama_claim_..."
}Claim tokens are opaque, stable, and bound to the signed-in Glama account. They contain no email address or other personal information. If Glama can no longer discover a verified HTTP or DNS token, it starts a seven-day grace period before removing claim-based access. Restore the same token during that period to keep ownership verified. Never publish an email address, Glama session token, GitHub token, or connector credential as ownership proof.
If verification fails, confirm that you copied the current token exactly. The HTTP file must be public, return valid JSON with a successful HTTP response, and stay on the connector's origin. DNS changes may need more time to propagate. A claim cannot transfer to a different origin or hostname: if the connector target changes, Glama starts the grace period and the new target must be claimed separately after the previous claim is released.
For a connector linked to the official MCP Registry, registry updates continue to replace its name, description, and URL by default. After claiming, open Manage connector and enable Use Glama listing details as the source of truth if edits made on Glama should be preserved. Categories and thumbnails are always managed on Glama; registry linkage and technical connection settings continue to sync.
Control your server's listing on Glama, including description and metadata
Access analytics and receive server usage reports
Get monitoring and health status updates for your server
Feature your server to boost visibility and reach more users
To improve your MCP server's ranking:
Claim ownership of the server listing
Complete the server profile with an accurate description and thumbnail
Provide a test profile so Glama can connect to and evaluate the server
Keep tool definitions clear and complete to earn a high Tool Definition Quality Score (TDQS)
Route real usage through the Glama Gateway; more recorded successful server uses also improve the ranking
For users:
Full audit trail – every tool call is logged with inputs and outputs for compliance and debugging
Granular tool control – enable or disable individual tools per connector to limit what your AI agents can do
Centralized credential management – store and rotate API keys and OAuth tokens in one place
Change alerts – get notified when a connector changes its schema, adds or removes tools, or updates tool definitions, so nothing breaks silently
For server owners:
Proven adoption – public usage metrics on your listing show real-world traction and build trust with prospective users
Tool-level analytics – see which tools are being used most, helping you prioritize development and documentation
Direct user feedback – users can report issues and suggest improvements through the listing, giving you a channel you would not have otherwise
The connector status is unhealthy when Glama is unable to successfully connect to the server. This can happen for several reasons:
The server is experiencing an outage
The URL of the server is wrong
Credentials required to access the server are missing or invalid
If you are the owner of this MCP connector and would like to make modifications to the listing, including providing test credentials for accessing the server, please contact support@glama.ai.
No comments yet. Be the first to start the discussion!
Checks llms.txt, AI crawler access in robots.txt, and sitemap - with a 0-100 AI readiness score.
Identify AI crawlers by user-agent or IP, verify them against operator ranges, build robots.txt.
Agent-readiness scanner (0-5 score), robots.txt + llms.txt generators, managed agent enablement.
Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.
