Collect articles from news sites by date range
collect_news_site_articlesRetrieve full articles published by named news sites within a past date range, including text, title, author, date, and section, using sitemaps with Internet Archive fallback for missing pages.
Instructions
Find the articles a named news site published in a past date range, from its sitemaps and the Internet Archive capture index, and return full text, published date, title, author, and section. Paywalled and refused pages return a free row with no text. No browser, so sites built by JavaScript return little. Pay per event on Apify: 0.003 USD per article.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| sites | No | News sites by domain, for example techcrunch.com, up to 20. | |
| date_to | No | The latest published date, as YYYY-MM-DD. Default: today. | |
| date_from | No | The earliest published date, as YYYY-MM-DD. Default: 30 days before Date to. | |
| min_words | No | Pages with fewer words are not counted as articles and are free. Default 80. | |
| obey_robots | No | yes (default): pages robots.txt disallows are skipped and counted. no: read them anyway. | |
| url_contains | No | Optional. Keep only addresses with any of these texts, for example /politics/. | |
| max_charge_usd | No | Optional. The run stops before a charge that would pass this amount. Summary rows are always returned. | |
| use_archive_copy | No | yes (default): a page the site answers with 404, 410, 500, 502, or 504 is read from the Internet Archive, at the archived row price. Refused and paywalled pages are never read from an archive. no: gone pages return a free row. | |
| use_archive_index | No | yes (default): when the sitemaps run out, also find addresses in the Internet Archive capture index, which reaches years that sitemaps no longer list. no: sitemaps only. | |
| max_articles_per_site | No | Articles to return from each site, up to 5,000. Default 100. |