Stop a running crawl by killing its worker process. Results so far are kept.

Check the site's robots.txt: is url crawlable, and what crawl-delay applies?

user_agent: the agent to evaluate rules for (defaults to the server's user agent). Returns {allowed, has_robots, crawl_delay, request_rate, sitemaps, robots_url}. Sites with no robots.txt are reported as allowed (has_robots=false).

Page through a crawl's scraped items.

cursor: line offset to start from (use the returned next_cursor to continue). limit: items per page (1-1000). Safe to call while the crawl is still running -- you get whatever has been written so far.

Report a crawl's state and progress.

state is one of: starting, running, finished, failed, cancelled, interrupted. ('interrupted' means the worker process died without finishing -- e.g. the machine or server restarted; any results gathered so far are still readable.)

Fetch a page and pull structured fields out of it with CSS or XPath selectors.

selectors maps an output field name to a selector. Each value is either:

• a CSS string, e.g. "h1::text" or "a.product::attr(href)" (returns the first match), or

• an object: {"css": "...", "all": true} / {"xpath": "//h1/text()", "all": true} ("all": true returns every match as a list; default returns the first match).

Example: {"title": "h1::text", "prices": {"css": ".price::text", "all": true}} Returns {"data": {field: value | [values]}}.

Fetch a page and return its links (de-duplicated, absolute URLs).

same_domain: keep only links on the same host as url. pattern: keep only links whose URL matches this regular expression. limit: maximum number of links to return (1-2000). Each link is {"url": ..., "text": ..., "nofollow": bool}.

Fetch a page and extract every HTML as {headers, rows}.

max_tables: cap on how many tables to return (1-100). Each table is {"headers": [...], "rows": [[cell, ...], ...], "n_rows": int, "n_cols": int}.

Fetch a single web page and return its content (no JavaScript rendering).

format: 'markdown' (default, compact and readable), 'text' (visible text only), or 'html' (raw HTML). max_bytes: cap on returned characters; defaults to the server's SCRAPY_MCP_MAX_BYTES. Oversized content is truncated and truncated is set to true. obey_robots: override the server's robots.txt policy for this call only.

An HTTP error status (404, 500, ...) is returned in status, not raised.

Fetch a sitemap and return the URLs it lists.

Handles gzip-compressed sitemaps and recurses one level of into its child sitemaps. url should point at a sitemap (e.g. https://site.com/sitemap.xml). limit: maximum URLs to return (1-50000). Each entry is {"loc": ..., "lastmod": ..., "changefreq": ..., "priority": ...}.

Start a bounded breadth-first crawl. Returns a job_id immediately (non-blocking).

The crawl follows in-scope links from start_url up to max_pages / max_depth.

• allow_patterns / deny_patterns: regexes a link URL must match / must not match.

• same_domain (default true): restrict the crawl to start_url's host.

• selectors: same format as the extract tool; when given, each crawled page yields the extracted data. Otherwise each page yields a short text excerpt.

• obey_robots, download_delay: per-crawl overrides of the server defaults.

Poll progress with crawl_status(job_id) and read items with crawl_results(job_id).