Manage an asynchronous crawl of authorized public pages on one website. Starting from one URL, the crawler follows same-domain links breadth-first and can also seed from the site's /sitemap.xml. Requests must comply with destination policy, robots directives, rate limits, and access controls.

Use this tool to discover pages for site audits, authorized retrieval worklists, or coverage checks before fetching individual pages. Not for: reading one page's content (use web_access_fetch) or open-ended web queries (use web_access_search).

This single tool drives the whole job lifecycle through action:

1. submit — quote a crawl (nothing is crawled or billed yet). Requires url. Optional: maxPages (1–10000, default 10), maxDepth (1–100, default 2), pathPrefix (only crawl URLs whose path starts with this, e.g. "/docs"), budgetUsd (spend ceiling; the crawl stops with status token_cap_exceeded if it would exceed it), useSitemap (also seed the site's root /sitemap.xml — one extra billed page, but finds pages links miss). A maxPages or maxDepth outside its range, 0 included, is rejected rather than replaced by the default — omit the field to get the default. Returns jobId, estimatedPages, and estimatedCostUsd with status awaiting_approval.

{ "action": "submit", "url": "https://example.com", "maxPages": 200, "maxDepth": 3 }

2. approve — start the quoted crawl (requires jobId). This is the billing-consent step: pages are billed as they are fetched, capped by the quote/budget. Before approving a non-trivial estimatedCostUsd, confirm the spend with your user. Fails with status 402 if the account balance cannot cover the quote; a 409 partial_state error means an earlier approve was interrupted — just call approve again.

3. status — poll progress (requires jobId). Statuses: awaiting_approval → running → terminal completed | failed | canceled | token_cap_exceeded (budget hit before maxPages; collected results are still readable). While running it returns pending and processed counts; a partial_state status means an interrupted approve — call approve again to repair it. Status never includes the URL list — page that with results. Poll every few seconds for small crawls; give hundreds-of-pages crawls tens of seconds between polls.

4. results — page through discovered URLs (requires jobId). Optional limit (1–5000, default 1000) and offset (≥ 0); out-of-range values are rejected. total tells you when to stop paging. Each entry has url, statusCode (0 = discovered but not fetched), depth, parentUrl, isSitemap, sourceType, and an error when that page failed. discoveredUrls (links found on the page) is only present for ~1h after completion; afterwards results come from durable storage which omits it — everything else stays available.

5. cancel — stop a running or pending job (requires jobId). Already-terminal jobs return a 409 error. Pages already fetched stay billed and readable via results.

6. list — recent crawl jobs for the account. Optional limit (1–100, default 20) and offset (≥ 0); out-of-range values are rejected. Use it to find a jobId you lost or check for an equivalent recent crawl before paying for a new one.

Typical workflow: submit → check estimatedCostUsd → approve → poll status until terminal → results (paged). A 404 on any jobId action means the job doesn't exist or belongs to another account; a 403 on submit means the target domain is blocked for this account (contact support@usestring.ai).

Returns: the JSON envelope for the chosen action (quote, status, URL page, job list) alongside a one-line summary.