seo-geo-mcp-server

robots.txt Check

robots_txt_check

Read-onlyIdempotent

Parses a site's robots.txt to reveal allow/disallow rules, declared sitemaps, and parse warnings. Flags wildcard disallow rules and 5xx responses that can silently deindex the site.

Instructions

Fetch and parse a site's robots.txt per RFC 9309. Reports every user-agent group with its Allow/Disallow rules, the declared sitemaps, and any lines that could not be parsed. Flags the two failures that silently deindex a site: a wildcard Disallow: /, and a robots.txt that returns 5xx (which Google treats as "disallow everything").

Args:

site (string): domain or any URL on it, e.g. 'example.com'.
response_format ('markdown' | 'json'): output format (default 'markdown').

Returns: { found, status, group_count, sitemaps[], blocks_everything, groups[{agents[], rules[], crawl_delay}], parse_warnings[], findings[] }.

Example: "What does example.com's robots.txt allow?" -> robots_txt_check(site="example.com"). For AI-crawler specifics use ai_crawler_access instead — it resolves each known AI bot against these rules.

Input Schema

TableJSON Schema

Name	Required	Description	Default
`site`	Yes	Site domain or any URL on it, e.g. 'example.com'. Only the origin is used.
`response_format`	No	Output format: 'markdown' for a human-readable summary (default) or 'json' for the full structured payload.	markdown

Output Schema

TableJSON Schema

Name	Required	Description	Default
`url`	Yes
`found`	Yes
`groups`	Yes
`status`	Yes
`findings`	Yes
`sitemaps`	Yes
`group_count`	Yes
`parse_warnings`	Yes
`blocks_everything`	Yes

Schema Changelog

Changes observed during successful MCP inspections.

Changed3 schema fields changedv1.2.0Sep 16, 2026

changedInput schema / $schema

Previous value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"

removedInput schema / additionalProperties
```
Removed value: -false
```

changedOutput schema / $schema

Previous value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"

First observedv1.0.0Aug 3, 2026

TDQS

A4.7/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, and non-destructive behavior. The description goes well beyond by disclosing exact parsing behavior (per RFC 9309), the special flags for `Disallow: /` and 5xx responses, and the structure of the returned payload. No contradictions with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Every sentence earns its place: purpose, special behavior flags, args, return shape, example, and sibling routing. The structure is logical and front-loaded with the core purpose before operational details.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with two params and rich annotations, the description includes everything needed to select and invoke it correctly: parameter usage, return fields, default behavior, and a clear alternative for a related but different task. The example further anchors correct invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already fully explains `site` and `response_format`. The description repeats the args and adds a concrete example mapping a natural-language query to a call, but does not add meaning beyond the schema's existing detail.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Starts with a specific verb and resource: 'Fetch and parse a site's robots.txt per RFC 9309'. It clearly defines the scope (every user-agent group, Allow/Disallow rules, sitemaps, unparseable lines) and distinguishes itself from sibling `ai_crawler_access` by naming that alternative explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides an example query ('What does example.com's robots.txt allow?') and an explicit when-not-to-use instruction: 'For AI-crawler specifics use ai_crawler_access instead'. It also signals a diagnostic use case by explaining the two silent deindexing failures it flags.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server