scrape_blog
Discover all posts on a blog and write them into one aggregated Markdown file. Uses RSS/Atom feeds or HTML crawling to find post URLs, then fetches and cleans each article's text.
Instructions
Discover every post on a blog and write a single aggregated Markdown file.
Tries the blog's RSS/Atom feed first (common paths like /rss.xml,
/feed, /atom.xml). If no feed responds, crawls the HTML index
(up to blog_scrape_discovery_pages pages). Each discovered post
URL is fetched in parallel and its cleaned main text is appended
to one Markdown file at:
{blog_scrape_output_dir}/<safe-host>_<safe-path>_<unix-ts>.md
The file starts with YAML front matter (blog URL, post count,
byte total, duration) followed by one H3 section per post with
its title, original URL, published date (if known from the feed),
content-type, byte count, and the cleaned text. Failures are
noted in-band as `- Fetch error: <code>` bullets, not by aborting
the whole run.
Args:
blog_url: Root URL of the blog (e.g. "https://blog.example.com/").
Returns:
Small summary dict — never the file contents:
{blog_url, output_path, post_count, post_count_ok, byte_count,
duration_s, discovery}
On failure: {blog_url, error: <stable_code>, message}.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| ctx | No | ||
| blog_url | Yes |
Output Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||