web_access_sitemap

Crawl an entire website and map its URLs using String AI's Web Access API sitemap crawler. Starting from one URL it follows same-domain links breadth-first (optionally seeded from the site's /sitemap.xml) and records every URL it reaches with fetch status, depth, and parent. The crawl runs asynchronously server-side, so it handles whole sites that a single web_access_fetch call cannot.

Best for: discovering all pages/URLs of a site (site audits, building scraping worklists, coverage checks) before fetching individual pages with web_access_fetch. Not for: reading one page's content (use web_access_fetch) or open-ended web queries (use web_access_search).

This single tool drives the whole job lifecycle through action:

1. submit — quote a crawl (nothing is crawled or billed yet). Requires url. Optional: maxPages (1–10000, default 10), maxDepth (1–100, default 2), pathPrefix (only crawl URLs whose path starts with this, e.g. "/docs"), budgetUsd (spend ceiling; the crawl stops with status token_cap_exceeded if it would exceed it), useSitemap (also seed the site's root /sitemap.xml — one extra billed page, but finds pages links miss). Returns jobId, estimatedPages, and estimatedCostUsd with status awaiting_approval.

{ "action": "submit", "url": "https://example.com", "maxPages": 200, "maxDepth": 3 }

2. approve — start the quoted crawl (requires jobId). This is the billing-consent step: pages are billed as they are fetched, capped by the quote/budget. Before approving a non-trivial estimatedCostUsd, confirm the spend with your user. Fails with status 402 if the account balance cannot cover the quote; a 409 partial_state error means an earlier approve was interrupted — just call approve again.

3. status — poll progress (requires jobId). Statuses: awaiting_approval → running → terminal completed | failed | canceled | token_cap_exceeded (budget hit before maxPages; collected results are still readable). While running it returns pending and processed counts; a partial_state status means an interrupted approve — call approve again to repair it. Status never includes the URL list — page that with results. Poll every few seconds for small crawls; give hundreds-of-pages crawls tens of seconds between polls.

4. results — page through discovered URLs (requires jobId). Optional limit (default 1000, max 5000) and offset; total tells you when to stop paging. Each entry has url, statusCode (0 = discovered but not fetched), depth, parentUrl, isSitemap, sourceType, and an error when that page failed. discoveredUrls (links found on the page) is only present for ~1h after completion; afterwards results come from durable storage which omits it — everything else stays available.

5. cancel — stop a running or pending job (requires jobId). Already-terminal jobs return a 409 error. Pages already fetched stay billed and readable via results.

6. list — recent crawl jobs for the account. Optional limit (default 20, max 100) and offset. Use it to find a jobId you lost or check for an equivalent recent crawl before paying for a new one.

Typical workflow: submit → check estimatedCostUsd → approve → poll status until terminal → results (paged). A 404 on any jobId action means the job doesn't exist or belongs to another account; a 403 on submit means the target domain is blocked for this account (contact support@usestring.ai).

Returns: the JSON envelope for the chosen action (quote, status, URL page, job list) alongside a one-line summary.