scrapy-mcp

by eitan3

Overview Schema Related Servers Score Discussions

Python

Local

Server Quality Checklist

Profile completionA complete profile improves this server's visibility in search results.

Latest release: v0.1.0
Disambiguation5/5
Each tool has a clearly distinct purpose: fetching a page, extracting tables, links, structured data, checking sitemap/robots, and managing crawls. There is no overlap between single-page extraction tools and crawl lifecycle tools.
Naming Consistency5/5
Tool names consistently follow a verb_noun pattern: extract_tables, extract_links, fetch_page, get_sitemap, check_robots, start_crawl, crawl_status, etc. The generic 'extract' is slightly less patterned but still readable and predictable.
Tool Count5/5
With 10 tools, the server is well-scoped for its purpose. The set balances single-page extraction utilities with essential crawl management operations without unnecessary bloat.
Completeness4/5
The domain of web scraping and crawling is well covered: page fetching, link/table/structured extraction, robots and sitemap handling, plus full crawl lifecycle (start, status, results, cancel). Minor gap is lack of a tool to list all crawls, but core workflows are complete.
Average 4.1/5 across 10 of 10 tools scored.
See the Tool Scores section below for per-tool breakdowns.
- No community issues in the last 6 months
- 1 commit in the last 12 weeks
- No stable releases found
- No critical vulnerability alerts
- No high-severity vulnerability alerts
- No code scanning findings
- CI status not available
This repository is licensed under MIT License.
This repository includes a README.md file.
No tool usage detected in the last 30 days. Usage tracking helps demonstrate server value.
Tip: use the "Try in Browser" feature on the server page to seed initial usage.
Add a glama.json file to provide metadata about your server.
If you are the author, simply .
If the server belongs to an organization, first add glama.json to the root of your repository:
```
{
  "$schema": "https://glama.ai/mcp/schemas/server.json",
  "maintainers": [
    "your-github-username"
  ]
}
```
Then . Browse examples.
Add related servers to improve discoverability.

How to sync the server with GitHub?

Servers are automatically synced at least once per day, but you can also sync manually at any time to instantly update the server profile.

To manually sync the server, click the "Sync Server" button in the MCP server admin interface.

How is the quality score calculated?

The overall quality score combines two components: Tool Definition Quality (70%) and Server Coherence (30%).

Tool Definition Quality measures how well each tool describes itself to AI agents. Every tool is scored 1–5 across six dimensions: Purpose Clarity (25%), Usage Guidelines (20%), Behavioral Transparency (20%), Parameter Semantics (15%), Conciseness & Structure (10%), and Contextual Completeness (10%). The server-level definition quality score is calculated as 60% mean TDQS + 40% minimum TDQS, so a single poorly described tool pulls the score down.

Server Coherence evaluates how well the tools work together as a set, scoring four dimensions equally: Disambiguation (can agents tell tools apart?), Naming Consistency, Tool Count Appropriateness, and Completeness (are there gaps in the tool surface?).

Tiers are derived from the overall score: A (≥3.5), B (≥3.0), C (≥2.0), D (≥1.0), F (<1.0). B and above is considered passing.

Tool Scores

Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the burden. It discloses the max_tables cap and specifies the return structure, but omits details like network side effects, robots.txt handling, error behavior, or whether JavaScript is executed. It is not misleading but lacks depth.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three sentences, front-loaded with the main purpose, and each sentence adds value. The return format is compactly specified, with no wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, the description does provide the return format, which is helpful. However, it lacks guidance on usage context, exclude cases, and the third param (obey_robots), making it only partially complete for a moderate-complexity tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With 0% schema description coverage, the description must compensate. It defines max_tables well (cap 1-100) and clarifies url implicitly via 'Fetch a page', but provides no meaning for obey_robots, which remains ambiguous. Partial compensation for a 3-param tool.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool fetches a page and extracts HTML <table> elements, with a specific output format. This distinguishes it from siblings like extract_links or fetch_page by focusing on tables as the resource.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for table extraction but does not explicitly mention when to use this tool over alternatives like fetch_page or extract. No exclusions or alternative tool names are provided, leaving usage guidance implicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses possible state values and explains the 'interrupted' state in detail, including that results remain readable. This adds meaningful behavioral insight beyond a basic status check, though error behavior is not covered.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and front-loaded with the core purpose, followed by a necessary list of state values and a clarifying note. No redundant wording.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is simple, but without an output schema the description leaves the return format ambiguous ('state and progress' but no structure). It also does not address invalid job IDs, though the state enumeration is a helpful addition.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has one required parameter, job_id, with no description beyond the name. The tool description does not mention job_id or its provenance (e.g., returned by start_crawl), leaving parameter semantics under-specified despite 0% schema coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Report a crawl's state and progress.' with a specific verb and resource. This distinguishes it from sibling tools like start_crawl and crawl_results, which focus on initiating and fetching results.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides context about the meaning of states, especially 'interrupted,' but does not explicitly state when to use this tool versus alternatives like crawl_results or cancel_crawl. Usage is implied but not explicitly articulated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden of disclosure. It does disclose meaningful behaviors: deduplication, absolute URL conversion, and the output link object structure. However, it omits any explanation of the `obey_robots` parameter, which could significantly affect behavior (e.g., respecting robots.txt), and provides no details on error handling or edge cases like pages with no links.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is efficiently structured: a one-sentence core purpose followed by a compact parameter list and output format. Every line adds value with no redundancy, using a clear code block for parameters and return shape. It is front-loaded with the essential 'fetch and return links' message.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite no annotations or output schema, the description provides a solid overview of the tool's behavior and return format. It explains the main parameters and output object. The primary gap is the undocumented `obey_robots` parameter and lack of mention of potential errors or page fetch failures, preventing a perfect score.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It does explain `same_domain` (same host), `pattern` (regular expression), and `limit` (1-2000), adding value beyond the schema. However, `obey_robots` is entirely absent, and `pattern` lacks specifics about regex flavor or matching rules. The description covers 3 of 5 parameters meaningfully but leaves key gaps.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The first sentence clearly states the tool's function: 'Fetch a page and return its links (de-duplicated, absolute URLs).' It uses a specific verb and resource, and the output is clearly defined. This distinguishes it from sibling tools like fetch_page (returns page content) and extract (generic extraction) by focusing solely on link extraction with deduplication and URL normalization.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage contexts through the parameter explanations (e.g., same_domain, pattern) but provides no explicit guidance on when to choose this tool over alternatives such as extract_tables or get_sitemap. There are no 'use this when...' or 'for X use...' statements, leaving usage as inferred rather than explicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It usefully explains that gzip-compressed sitemaps are handled, that `<sitemapindex>` is recursed one level, and the exact output entry structure. This adds non-obvious context beyond what the schema would imply.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is appropriately concise, with the purpose stated in the first sentence and additional details organized clearly. Every sentence adds value, and there is no redundant or fluff content.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple fetch tool with three parameters and no output schema, the description covers the main behaviors (gzip handling, index recursion, output format) and gives a working example. It omits details about `obey_robots` behavior and error cases, but overall it is substantially complete for common usage.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It does explain `url` and `limit` meaningfully (limit with a 1-50000 range), but `obey_robots` is entirely undocumented. Since one of three parameters is unexplained, the compensation is incomplete.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's function: 'Fetch a sitemap and return the URLs it lists.' The verb 'Fetch' and resource 'sitemap' are specific, and the output (URLs) is identified. It distinguishes itself from sibling tools by focusing exclusively on sitemaps rather than general pages or link extraction.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides a clear prerequisite: '`url` should point at a sitemap (e.g. https://site.com/sitemap.xml).' It also mentions the limit parameter. However, it does not explicitly compare to alternative tools like fetch_page or extract_links, nor does it state when not to use this tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses the destructive act of killing the worker process and the side effect that results are kept. This goes beyond a vague 'cancel' and gives the agent meaningful expectations about consequences.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two short sentences, front-loaded with the core action and immediately followed by the most important side effect. No filler or redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (one parameter, no output schema, no annotations), the description covers the key aspects: what it stops, how it stops it, and what persists. It doesn't mention whether cancellation is reversible or what is returned, but those are not critical for this simple operation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has one required parameter (job_id) but zero description coverage. The description does not explicitly define job_id, though the tool name and context make it inferable as the crawl's ID. It would benefit from stating 'the ID of the crawl to cancel' or referencing start_crawl.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action (stop a running crawl), the mechanism (killing its worker process), and the outcome (results kept). This is a specific verb+resource+scope statement that distinguishes cancel_crawl from siblings like start_crawl and crawl_status.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies the use case: when a crawl is running and must be stopped. It does not explicitly contrast with alternatives, but the purpose is so singular that the context is clear. It would benefit from mentioning 'use crawl_status to check if a crawl is running', but the baseline is solid.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description discloses the return fields and the special behavior for missing robots.txt (reported as allowed with has_robots=false). Since there are no annotations, this carries the full burden and effectively communicates the tool's read-only nature and output.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise: one line for purpose, one for parameter, and one for return values. It is front-loaded with the main purpose and contains no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple read-only tool with no output schema, the description fully covers: purpose, parameter semantics, return object fields, and an edge case (missing robots.txt). It is complete and self-contained.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description compensates by explaining user_agent's default and role. The url parameter is implicitly described as the site to check but lacks explicit format details; overall it adds meaning beyond the raw schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool checks a site's robots.txt to determine if a URL is crawlable and what crawl-delay applies. This is a specific verb+resource that distinguishes it from siblings like fetch_page or get_sitemap.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The tool's purpose implies it should be used before crawling to check compliance, but there is no explicit 'when to use' or 'when not to use' guidance. Siblings like start_crawl exist but no alternative is mentioned, leaving usage inferred rather than stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description fully carries the transparency burden. It clearly discloses the lack of JavaScript rendering, truncation behavior with the 'truncated' flag, robots.txt override behavior, and that HTTP errors are returned in 'status' rather than raised. This is comprehensive and goes beyond basic expectations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the core purpose in the first sentence, followed by a concise bulleted parameter list and a final sentence on error handling. Every sentence adds value with no fluff or redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers the main behavior (content fetching, format options, truncation, error handling) and notes the 'status' and 'truncated' fields. However, it does not provide a full return structure or mention any response headers/URL that might be included, leaving some ambiguity about the exact response shape.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters5/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema description coverage is 0%, so the description must fully compensate. It explains 'format', 'max_bytes', and 'obey_robots' in detail, including defaults and effects. The 'url' parameter is implicitly clear from the tool's purpose. This exceeds the schema's bare definitions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: 'Fetch a single web page and return its content'. It is clear and concise, but it does not explicitly differentiate from sibling tools like 'extract' or 'extract_links'. The 'no JavaScript rendering' note hints at scope but is not a direct alternative comparison.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for retrieving raw page content, but it does not explicitly state when to use this tool over siblings, nor does it provide exclusions (e.g., 'for structured data use extract'). The parameter explanations offer context but not direct comparative guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It discloses key behavioral traits: cursor is a line offset, next_cursor is used to continue, limit range (1-1000), and that results reflect whatever has been written so far during an active crawl. This adds valuable context beyond the schema and clarifies the tool's eventual-consistency behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is brief and front-loads the main purpose. It then uses a clear list-style block for parameter guidance. Every sentence adds value, with no filler or repetition. The formatting is clean and readable.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple pagination tool with three parameters and no output schema, the description covers the primary purpose, parameter semantics, and a key behavioral nuance. It hints at the response shape via 'next_cursor' but does not explicitly describe the structure of items. Overall, this is complete enough for effective tool invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has 0% description coverage, so the description compensates well by explaining cursor and limit in detail. It does not explicitly describe job_id, but the tool name and 'a crawl's' context make it obvious that job_id identifies the crawl. This is sufficient for a required identifier parameter.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb "Page through" and clearly identifies the resource as "a crawl's scraped items." This distinguishes it from sibling tools like start_crawl or crawl_status, making its function unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context on when to use the tool (to retrieve crawl results in pages) and explains the pagination mechanism via cursor and limit. It also notes it can be safely used while the crawl is running, which implies appropriate usage during execution. It does not explicitly exclude alternatives, but the context is sufficient.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the transparency burden. It explains selector syntax, first-match vs all behavior, and return format. However, it does not disclose the meaning of the obey_robots parameter or potential error handling, leaving some behavioral aspects untold.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-organized and front-loaded with the main purpose, followed by a compact specification of selector formats and an example. Every section adds necessary value, and formatting uses clear bullet points and code blocks.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description is largely complete for a complex extraction tool, including return structure and selector behavior. It could be more complete by addressing edge cases, obey_robots semantics, or error handling, but these are secondary to the core extraction logic.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It does so thoroughly for the selectors parameter, detailing CSS string and object forms with examples. The url parameter is implied by 'Fetch a page.' The obey_robots parameter is not explained, but its name and default null convey partial meaning.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's action: 'Fetch a page and pull structured fields out of it with CSS or XPath selectors.' It specifies the resource (page) and the mechanism (CSS/XPath), distinguishing it from siblings like extract_tables and extract_links which are more specialized.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context on how to use the tool by explaining the selectors mapping and examples. However, it does not explicitly mention when not to use it or alternatives such as extract_tables or fetch_page, so it falls short of full guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of disclosing behavior. It covers the non-blocking nature, bounded limits, same_domain default, robots/delay overrides, and selector semantics. It does not mention cancellation via cancel_crawl or potential rate limits, but the disclosed details are substantial and contextually useful.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise, front-loaded with the most important behavioral fact (non-blocking, returns job_id), and uses bullet-like separations for parameters. Every sentence adds value, and the overall length is appropriate for the tool's complexity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a complex crawl tool with no output schema, the description covers the initiation, parameter semantics, and follow-up tools, which is largely complete. It omits details about cancellation or error handling, but the presence of sibling tools like crawl_status and cancel_crawl partially fills that gap. Given the high complexity, a 4 is appropriate.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters5/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate for all 9 parameters. It explains allow/deny_patterns as regex constraints, same_domain as host restriction, selectors as same format as extract tool, and obey_robots/download_delay as per-crawl overrides. This fully covers the non-obvious parameters, leaving little ambiguity.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool starts a bounded breadth-first crawl and immediately returns a job_id, using a specific verb and resource. It distinguishes itself from sibling tools like crawl_status and crawl_results by focusing on initiation rather than status checking or retrieval.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context on how the crawl behaves and explicitly instructs to poll with crawl_status and read results with crawl_results. It does not explicitly contrast with alternatives like fetch_page or get_sitemap, but the follow-up workflow is clearly indicated, earning a 4 rather than a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

GitHub Badge

Glama performs regular codebase and documentation scans to:

Confirm that the MCP server is working as expected.
Confirm that there are no obvious security issues.
Evaluate tool definition quality.

Our badge communicates server capabilities, safety, and installation instructions.

Card Badge

Copy to your README.md:

[![Scrapy_MCP_Scraper MCP server](https://glama.ai/mcp/servers/eitan3/Scrapy_MCP_Scraper/badges/card.svg)](https://glama.ai/mcp/servers/eitan3/Scrapy_MCP_Scraper)

Score Badge

Copy to your README.md:

[![Scrapy_MCP_Scraper MCP server](https://glama.ai/mcp/servers/eitan3/Scrapy_MCP_Scraper/badges/score.svg)](https://glama.ai/mcp/servers/eitan3/Scrapy_MCP_Scraper)

Latest Blog Posts

Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)
By Om-Shree-0709 on July 25, 2026.
mcp
Agent Identity
OAuth 2.1
Your AI Chatbot Just Exposed Your CEO's Salary to an Intern
By Om-Shree-0709 on July 2, 2026.
Agent Identity
MCP Security
OAuth Delegation
Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)
By Om-Shree-0709 on June 30, 2026.
Agentic Ai
Prompt Injection
WebAssembly

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/eitan3/Scrapy_MCP_Scraper'

If you have feedback or need assistance with the MCP directory API, please join our Discord server