Skip to main content
Glama
Sign In
scrapy-mcp

scrapy-mcp

by eitan3
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Local

get_sitemap

Retrieve all URLs from a sitemap, including gzip-compressed and nested sitemap indexes, with metadata such as lastmod and priority.

Instructions

Fetch a sitemap and return the URLs it lists.

Handles gzip-compressed sitemaps and recurses one level of into its child sitemaps. url should point at a sitemap (e.g. https://site.com/sitemap.xml). limit: maximum URLs to return (1-50000). Each entry is {"loc": ..., "lastmod": ..., "changefreq": ..., "priority": ...}.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>"
    },
    "name": "get_sitemap"
  }
}

Input Schema

NameRequiredDescriptionDefault
urlYes
limitNo
obey_robotsNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.1.0

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It usefully explains that gzip-compressed sitemaps are handled, that `<sitemapindex>` is recursed one level, and the exact output entry structure. This adds non-obvious context beyond what the schema would imply.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is appropriately concise, with the purpose stated in the first sentence and additional details organized clearly. Every sentence adds value, and there is no redundant or fluff content.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple fetch tool with three parameters and no output schema, the description covers the main behaviors (gzip handling, index recursion, output format) and gives a working example. It omits details about `obey_robots` behavior and error cases, but overall it is substantially complete for common usage.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It does explain `url` and `limit` meaningfully (limit with a 1-50000 range), but `obey_robots` is entirely undocumented. Since one of three parameters is unexplained, the compensation is incomplete.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's function: 'Fetch a sitemap and return the URLs it lists.' The verb 'Fetch' and resource 'sitemap' are specific, and the output (URLs) is identified. It distinguishes itself from sibling tools by focusing exclusively on sitemaps rather than general pages or link extraction.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides a clear prerequisite: '`url` should point at a sitemap (e.g. https://site.com/sitemap.xml).' It also mentions the limit parameter. However, it does not explicitly compare to alternative tools like fetch_page or extract_links, nor does it state when not to use this tool.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools