scrapewise_run_scraper_sitemaps_harvest
Harvests the scraper's target site's robots.txt + sitemap.xml (and any
referenced sub-sitemaps), persisting up to `cap` URLs as sitemap entries
attached to this scraper. Use to bootstrap a Site URL inventory from a public
sitemap before configuring per-page scraping. All sitemap entries are wiped
every night at 02:00 UTC — re-harvest daily if you need fresh state.
Billing: every robots.txt / sitemap file fetched is charged as one plain page
(free requests are used first). Re-harvesting a domain that already has entries
returns the stored count and fetches (and charges) nothing.
Limits: at most 500 files fetched, 5 levels of sitemap-index nesting, 500000 URLs
and 50 MiB per file (server-configured), and one harvest at a time per account.
When a limit or the wallet balance stops the harvest early the count is PARTIAL
and the response carries `X-Harvest-Limit-Reached` (comma-separated:
MAX_FILES, MAX_DEPTH, MAX_URLS, MAX_BYTES, WALLET_BUDGET) plus
`X-Harvest-Files-Fetched`.
Args:
- scraperId (string, path param, required): the scraper's MongoDB ObjectId.
Sitemap is fetched from the URL configured on this scraper.
- cap (int, query param, optional, default 500000): maximum number of URLs
to ingest. It can only lower the server limit, never raise it.
Returns:
`Long` — count of URLs ingested. 0 means the target site's sitemap.xml is
missing / unreachable / empty.
Examples:
- Use when: "ingest scraper {id}'s sitemap so I can browse its URLs"
- Use when: "ingest just the first 1000 URLs" → cap=1000
- Don't use when: you already have URLs from a Site → call
scrapewise_create_scraper_site directly with the link list
- Don't use when: the automation is a dedicated endpoint — paste a list or feed
from another automation
Error Handling:
- 402 WALLET_INSUFFICIENT_BALANCE when the wallet has no balance and no free
requests left — nothing is fetched.
- 429 if a harvest is already running for this account.
- 400 (CustomerError envelope) if no scraper with the given id exists for this customer.
- 502 if the sitemap.xml URL is unreachable or malformed.
- 401 Unauthorized.[runtime hint] This tool returns JSON-as-text — its response shape is dynamic (per-scraper itemsConfig, file binary, etc.) so the MCP outputSchema is omitted. Do not rely on the presence of any specific field in the response; treat the result as a string-or-object blob and parse defensively.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"scraperId": "<scraperId>"
},
"name": "scrapewise_run_scraper_sitemaps_harvest"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| cap | No | Optional maximum number of entries to harvest. Defaults to maximum limit witch is set to 500000. Useful to limit memory usage for very large sites. | |
| scraperId | Yes | the scraper's MongoDB ObjectId. Sitemap is fetched from the URL configured on this scraper. |