Crawl a website toward a goal and return only the relevant passages, verbatim.

Use this instead of fetching pages yourself when the answer may be several pages deep (documentation, help centers, policies, product catalogs). A small decision model scores every block of text and every link, follows the most promising links first, and stops as soon as the collected text answers the goal. No language model runs inside the crawl.

Args: goal: A specific question to answer, e.g. "How many days do I have to get a refund?". Vague goals ("learn about the shop") return vague results. start_url: Where to start; the crawl stays on this host and its subdomains. max_pages: Page limit. Start with 10-30. max_seconds: Wall-clock limit. return_tokens: Most text (estimated tokens) to hand back. max_decision_usd: Hard limit on what the decision model may cost for this call.

Returns: blocks: relevant passages, best first. Each has url, heading_path, text (copied from the page), char_start/char_end (offsets into the page text whose sha256 is in pages) and relevance (0-1). uncertain: borderline passages. Read these before concluding something is missing. pages: every page fetched. stats: what it cost and why it stopped; check stats.stopped_because ("answered", "frontier_empty", "max_pages", "max_seconds", "budget") and stats.degraded (true means the decision model was unreachable and results are unscored). stats.needs_js_urls lists pages that only render with JavaScript and could not be rendered; stats.js_rendered_urls those it rendered. On failure: {"error": code, "message": ...}.

Quote returned text together with its url.

Fetch one known page and return its text as blocks, optionally pruned to a goal.

Without goal: every block of the page that survives the free filters (navigation, footers and repeated boilerplate removed), in page order, with no decision-model call. With goal: only the blocks that help answer it (blocks) plus borderline ones (uncertain); links are not followed.

Returns the same shape as crawl (blocks, uncertain, pages, stats), or {"error": code, "message": ...}. Text is verbatim with char_start/char_end offsets.

Rank the URLs of a site by how likely they are to answer a goal, without returning content.

Runs the same best-first exploration as crawl but only scores links. Use it to decide which pages deserve a scrape, or to see what a site offers on a topic.

Returns urls: the top limit entries, best first, each {url, priority, p_target, p_hub, anchor, found_on} (p_target: the page itself likely answers the goal; p_hub: it likely leads to pages that do), plus pages (fetched) and stats. On failure: {"error": code, "message": ...}.