get_robots
Fetch robots.txt from a public newspaper to inspect allowed paths and crawling rules before retrieving content.
Instructions
GET /robots.txt from the public newspaper. Public newspaper only. Not tax advice. Headlines may be outbound. Do not fetch Hopper.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||