Skip to main content
Glama

scrapewise_run_scraper_sitemaps_harvest

        Harvests the scraper's target site's robots.txt + sitemap.xml (and any
        referenced sub-sitemaps), persisting up to `cap` URLs as sitemap entries
        attached to this scraper. Use to bootstrap a Site URL inventory from a public
        sitemap before configuring per-page scraping. All sitemap entries are wiped
        every night at 02:00 UTC — re-harvest daily if you need fresh state.

        Billing: every robots.txt / sitemap file fetched is charged as one plain page
        (free requests are used first). Re-harvesting a domain that already has entries
        returns the stored count and fetches (and charges) nothing.

        Limits: at most 500 files fetched, 5 levels of sitemap-index nesting, 500000 URLs
        and 50 MiB per file (server-configured), and one harvest at a time per account.
        When a limit or the wallet balance stops the harvest early the count is PARTIAL
        and the response carries `X-Harvest-Limit-Reached` (comma-separated:
        MAX_FILES, MAX_DEPTH, MAX_URLS, MAX_BYTES, WALLET_BUDGET) plus
        `X-Harvest-Files-Fetched`.

        Args:
          - scraperId (string, path param, required): the scraper's MongoDB ObjectId.
            Sitemap is fetched from the URL configured on this scraper.
          - cap (int, query param, optional, default 500000): maximum number of URLs
            to ingest. It can only lower the server limit, never raise it.

        Returns:
          `Long` — count of URLs ingested. 0 means the target site's sitemap.xml is
          missing / unreachable / empty.

        Examples:
          - Use when: "ingest scraper {id}'s sitemap so I can browse its URLs"
          - Use when: "ingest just the first 1000 URLs" → cap=1000
          - Don't use when: you already have URLs from a Site → call
            scrapewise_create_scraper_site directly with the link list
          - Don't use when: the automation is a dedicated endpoint — paste a list or feed
            from another automation

        Error Handling:
          - 402 WALLET_INSUFFICIENT_BALANCE when the wallet has no balance and no free
            requests left — nothing is fetched.
          - 429 if a harvest is already running for this account.
          - 400 (CustomerError envelope) if no scraper with the given id exists for this customer.
          - 502 if the sitemap.xml URL is unreachable or malformed.
          - 401 Unauthorized.

[runtime hint] This tool returns JSON-as-text — its response shape is dynamic (per-scraper itemsConfig, file binary, etc.) so the MCP outputSchema is omitted. Do not rely on the presence of any specific field in the response; treat the result as a string-or-object blob and parse defensively.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "scraperId": "<scraperId>"
    },
    "name": "scrapewise_run_scraper_sitemaps_harvest"
  }
}

Input Schema

NameRequiredDescriptionDefault
capNoOptional maximum number of entries to harvest. Defaults to maximum limit witch is set to 500000. Useful to limit memory usage for very large sites.
scraperIdYesthe scraper's MongoDB ObjectId. Sitemap is fetched from the URL configured on this scraper.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changed
    • addedInput schema / properties / scraperId / description
      Added value: +"the scraper's MongoDB ObjectId. Sitemap is fetched from the URL configured on this scraper."
  2. Added

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Goes far beyond the annotations: billing semantics (each file fetched charged as one page, free requests used first), the no-op short-circuit when a domain already has entries, nightly 02:00 UTC wipe of all sitemap entries, concurrency limit (one harvest per account), and machine-readable partial-result signals (X-Harvest-Limit-Reached reasons, X-Harvest-Files-Fetched). Error codes 402/429/400/502/401 are enumerated with their causes.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the action, then cleanly sectioned (Billing, Limits, Args, Returns, Examples, Error Handling). Slight redundancy: the Args section restates schema descriptions verbatim, and the returned count is explained both under Returns and again inside the partial-result header discussion.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a network-fetching, write-behind-cache mutation tool with no output schema, the definition supplies everything an agent needs: cost model, rate/concurrency limits, truncation signals, the 0-return meaning, and full error taxonomy. The runtime-hint caveat about dynamic response shape is the only remaining uncertainty and is external to the description.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so baseline is 3; the description adds genuinely useful semantics — that `cap` can only lower the server-side limit, never raise it, and that scraperId is a MongoDB ObjectId whose sitemap URL is taken from the scraper config. It adds modestly rather than substantially beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Opens with a specific verb+resource: harvests robots.txt + sitemap.xml (and sub-sitemaps) and persists up to `cap` URLs as sitemap entries attached to a named scraper. The scope is unambiguous and clearly distinct from sibling tools like scrapewise_get_scraper_sitemaps (read) and scrapewise_create_scraper_site (link list).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicit 'Use when' examples (bootstrap a Site URL inventory, cap to 1000 URLs) and two 'Don't use when' cases naming the correct alternatives: scrapewise_create_scraper_site with a link list, and the dedicated-endpoint automation path. Nothing is left to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.