Extract several pages
extract_manyExtract structured data from up to 25 URLs in order, returning summaries, records, and induced fields while respecting robots.txt and TDM rights.
Instructions
Read several pages' declared data, politely, in the order given.
urls: 1 to 25 http(s) addresses, on one site or several; one given twice is read once. records: also return each page's records, not only its summary; the heaviest pages' records are left out first to keep the answer under 75,000 bytes, each counted in that page's records_left_out. induce: also read repeated rows from a page that declares nothing about them; those fields say source "induced". respect_tdm: give a page whose site reserves its text and data mining rights (TDMRep) as a tdm_reserved error, never its data.
Returns {"ok", "pages", "stopped"}. Pages come in the order given, each {"ok", "url", "landed", "fetch", "canonical", "summary", "sources", "types", "links"}, and "records" when asked -- or, when it has nothing, {"ok": false, "error"} with the page's reason. A page asked again after a request that may succeed later says so in "retries". Each site is asked one request at a time, a second apart or its Crawl-delay, robots.txt obeyed; several sites at once. stopped is "done", or "time_budget" when a minute passed first and the pages after are left out. ok is false only when no page could be read, and error then says why. Past 75,000 bytes the heaviest pages' records go first, then the heaviest summary answers, named in summary_left_out, then the last pages, counted in pages_left_out. For many more addresses, or a whole site, the command line's sluicer batch has no such bounds.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| urls | Yes | 1 to 25 http(s) addresses, on one site or several; one given twice is read once. | |
| induce | No | also read repeated rows from a page that declares nothing about them; those fields say source "induced". | |
| records | No | also return each page's records, not only its summary; the heaviest pages' records are left out first to keep the answer under 75,000 bytes, each counted in that page's records_left_out. | |
| respect_tdm | No | give a page whose site reserves its text and data mining rights (TDMRep) as a tdm_reserved error, never its data. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| ok | Yes | ||
| error | No | ||
| pages | No | ||
| stopped | No | ||
| pages_left_out | No |