Skip to main content
Glama
Sign In
scrapy-mcp

scrapy-mcp

by eitan3
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Local

check_robots

Determines if a URL is crawlable by checking robots.txt, and returns the applicable crawl-delay, request rate, and sitemaps. Reports sites without robots.txt as allowed.

Instructions

Check the site's robots.txt: is url crawlable, and what crawl-delay applies?

user_agent: the agent to evaluate rules for (defaults to the server's user agent). Returns {allowed, has_robots, crawl_delay, request_rate, sitemaps, robots_url}. Sites with no robots.txt are reported as allowed (has_robots=false).

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>"
    },
    "name": "check_robots"
  }
}

Input Schema

NameRequiredDescriptionDefault
urlYes
user_agentNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.1.0

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description discloses the return fields and the special behavior for missing robots.txt (reported as allowed with has_robots=false). Since there are no annotations, this carries the full burden and effectively communicates the tool's read-only nature and output.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise: one line for purpose, one for parameter, and one for return values. It is front-loaded with the main purpose and contains no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple read-only tool with no output schema, the description fully covers: purpose, parameter semantics, return object fields, and an edge case (missing robots.txt). It is complete and self-contained.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description compensates by explaining user_agent's default and role. The url parameter is implicitly described as the site to check but lacks explicit format details; overall it adds meaning beyond the raw schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool checks a site's robots.txt to determine if a URL is crawlable and what crawl-delay applies. This is a specific verb+resource that distinguishes it from siblings like fetch_page or get_sitemap.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The tool's purpose implies it should be used before crawling to check compliance, but there is no explicit 'when to use' or 'when not to use' guidance. Siblings like start_crawl exist but no alternative is mentioned, leaving usage inferred rather than stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools