Read a page as markdown
page_markdownConvert any web page's main content to markdown, stripping navigation and footers. Fetch by URL or HTML, with options for page metadata and Wayback snapshots.
Instructions
Return a page's main content as markdown, without navigation or footer.
html_or_url: an http(s) URL to fetch, or the HTML itself. front_matter: open the markdown with a YAML block of what the page declares about itself (title, author, dates, url...) and each answer's source. at: a date: read the URL as the Wayback Machine captured it then. respect_tdm: answer tdm_reserved when the site reserves its text and data mining rights (TDMRep). offset: where in the markdown this answer starts, 0 for the beginning; the answer before gives the next as next_offset. max_chars: the most characters this answer carries, 1 to 60,000; fewer when more would weigh over 75,000 bytes, as 60,000 characters of Chinese do.
Returns {"ok", "markdown", "url", "length", "next_offset"}, and "fetch" for a URL: markdown is one slice, length the whole markdown's, next_offset where the next slice starts or null at the end. The markdown is always the page's own content: a failure is ok false with "error", never text that could be mistaken for the page.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| at | No | a date: read the URL as the Wayback Machine captured it then. | |
| offset | No | where in the markdown this answer starts, 0 for the beginning; the answer before gives the next as next_offset. | |
| max_chars | No | the most characters this answer carries, 1 to 60,000; fewer when more would weigh over 75,000 bytes, as 60,000 characters of Chinese do. | |
| html_or_url | Yes | an http(s) URL to fetch, or the HTML itself. | |
| respect_tdm | No | answer tdm_reserved when the site reserves its text and data mining rights (TDMRep). | |
| front_matter | No | open the markdown with a YAML block of what the page declares about itself (title, author, dates, url...) and each answer's source. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| ok | Yes | ||
| url | No | ||
| error | No | ||
| fetch | No | ||
| length | No | ||
| markdown | No | ||
| next_offset | No |