Skip to main content
Glama
Sign In
technical-seo-mcp

technical-seo-mcp

by sourav2024
OverviewSchemaSourceRelated ServersDiscussionsAdmin
JavaScript
Local

Check robots.txt

check_robots
Read-only

Fetch and validate /robots.txt to ensure it exists, is not an HTML page or redirect, references a sitemap, and does not block the entire site.

Instructions

Fetch /robots.txt, verify it exists and is valid (not an HTML page or redirect), references the sitemap, and does not block the entire site.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>"
    },
    "name": "check_robots"
  }
}

Input Schema

NameRequiredDescriptionDefault
urlYesAny URL on the site — the origin's /robots.txt is checked

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv1.0.0

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only declare readOnlyHint and openWorldHint. The description adds concrete behavioral details: fetching /robots.txt, checking for HTML/redirects, sitemap reference, and site-wide block. This goes beyond the annotations and gives the agent a clear picture of the operation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, well-structured sentence that front-loads the action ('Fetch') and lists all checks without redundancy. Every phrase adds information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a single-parameter tool with no output schema and read-only annotations, the description is quite complete. It covers what is fetched and the validation logic, though it doesn't describe the return format, which is a minor gap.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema already provides 100% coverage with the description 'Any URL on the site — the origin's /robots.txt is checked'. The tool description adds no further parameter detail, so baseline score of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb 'Fetch' and resource '/robots.txt', with explicit verification criteria (existence, not HTML/redirect, sitemap reference, no site-wide block). This clearly distinguishes it from sibling tools like check_sitemap and check_ai_crawlers.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description clearly implies the tool is for validating robots.txt, but it does not explicitly state when to use it versus alternatives like check_sitemap. However, the purpose is unambiguous enough that an agent would know when to invoke it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools