web_fetch
Read-only
Fetch a URL and return its main content as Markdown, PDF, or structured rows. Use query, CSS selectors, or token limits to extract only relevant blocks, tables, or data.
Instructions
Fetch one URL and return its main content as markdown (PDF and office documents too), or as rows with schema or tables. The default way to read a page: mode=auto climbs http -> browser -> stealth -> real -> warm, remembers the working tier per domain, and is never set by hand. Cut tokens with query (keep only relevant blocks), css_selector, max_tokens + cursor (page through), or out_file (write to disk, return a path). To click use web_snapshot; for the site's JSON API use web_capture; for several known URLs use web_fetch_many. Returns content, title, tier_used and quality; links to the page's own site are paths, to be joined to url. A wall returns blocked_reason and a note saying what to do: act on it, never retry blindly.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The page. Also `raw:<html>` for markup you already have (no request is made) and `file:///path` for a local file under `~/.svipall/in` or a configured `local_roots` entry. | |
| body | No | Request body for POST/PUT (http tier only). | |
| mode | No | Leave unset: auto learns the tier per domain. Forcing one (http, browser, stealth, real, warm) is for debugging and is slower or weaker. | |
| cache | No | auto (default: serve a fresh copy, revalidate a stale one), read, write, bypass, refresh. | |
| proxy | No | Proxy URL for this fetch only; web_route sets one per domain. | |
| query | No | Keep only the blocks relevant to these words (BM25), e.g. "shipping costs". The cheapest way to read a long page for one fact, and how much it saves is decided by the query, not the page: measured on two long articles, "robots.txt" and "caching headers" left 6% and 13% of the page, while "history of scraping" left 88%. Name the fact, not the topic. | |
| cursor | No | Continue a truncated response from where it stopped: the `cursor` of the previous result. | |
| method | No | HTTP method for the http tier: GET (default), POST, PUT, DELETE, HEAD. | |
| mobile | No | Ask as a phone: phone identity and viewport. Only worth it where a site serves a lighter page to phones — a responsive site, which is most of them, returns the same bytes: measured byte-identical on two sites at both the http and browser tiers. It also costs a browser page of its own, since no warm page is reused, and rules out the native last resort. | |
| robots | No | robots.txt policy: warn (default: fetch, and say whether robots.txt disallows it), obey (refuse a disallowed URL), ignore. | |
| schema | No | Rows instead of prose, returned as `extracted`. E.g. `{"name": "products", "base_selector": "div.product", "fields": [{"name": "title", "selector": "h2 a"}, {"name": "url", "selector": "a", "type": "attribute", "attribute": "href"}]}`; types text (default), attribute, number, exists, list, html, markdown. A named schema is remembered per domain, and a selector a redesign breaks is relocated and reported as `healed`. | |
| scroll | No | Scroll a page that loads as you go before reading it: "auto" until it stops growing (up to 40 screens, one "load more" click), or a number of rounds. Implies a browser tier. | |
| tables | No | Return the page's data tables as typed rows, `tables: [{caption, header, rows}]`, instead of prose: a fraction of the markdown, columns kept. Layout tables are skipped. | |
| headers | No | Extra request headers (http tier only). | |
| profile | No | Profile saved by web_login whose cookies to use. Implies a browser tier. | |
| timeout | No | Timeout in ms for the whole ladder. Default 60000. | |
| isolated | No | A throwaway browser profile for this fetch alone: no cookies in, nothing left behind. | |
| max_tier | No | Highest tier the ladder may climb to: http, browser, stealth, real, warm (default). | |
| out_file | No | Write the content to this file and return the path instead: measured, a 418-character response against the 34 746 characters of the page it wrote. Relative paths land in ~/.svipall/out/. With `schema` or `tables` a .csv, .json or .jsonl name writes the rows in that format. | |
| text_only | No | Skip images, fonts, stylesheets and video in browser tiers. Faster on heavy pages; off by default because some anti-bot scripts notice a page whose images never loaded. | |
| extraction | No | markdown (default), text, or html (the raw markup, many times the tokens). | |
| max_tokens | No | Cap on the content returned, cut on block boundaries. Default 25000. A truncated result carries a `cursor`. | |
| css_selector | No | Keep only the elements this CSS selector matches, e.g. "article" or "#prices". | |
| include_links | No | Add `links`, split into internal and external, plus images. | |
| include_quality | No | Add `quality_detail`: integrity verdict with reasons, optimisation traits, near-duplicates in the cache, provenance (byline, date, citations). For judging a source; the compact `quality` field is always present. | |
| include_metadata | No | Add `metadata`: canonical URL, language, author, dates, OpenGraph, JSON-LD, feeds. | |
| main_content_only | No | Drop navigation, footers and sidebars. Default true; false returns the whole body. | |
| use_site_template | No | Strip what this site repeats on every page (banners, footers), learned from earlier fetches of the same site. Off by default: it can take a word of real content with it. A response it changed says `"template": {"learned_from": 16, "removed_blocks": 3}`. |