deep_scrape
Crawl a blog to fetch every post, then download and extract text from all linked documents (PDF, DOCX, etc.) into one Markdown file.
Instructions
Discover every post on a blog, fetch each, AND download + extract every linked document (PDF/DOCX/XLSX/PPTX/CSV/JSON/XML/MD/EPUB/...).
Same discovery as `scrape_blog` (RSS/Atom first, HTML index
fallback). For every post fetched, the raw HTML is scanned for
links whose URL ends in a recognized document extension. Each
such document is downloaded (capped by
`deep_scrape_max_documents_per_blog`, 100 by default) and its
text is extracted via markitdown. The original blog post text
and every document's extracted text are all inlined into one
Markdown file at:
{blog_scrape_output_dir}/<safe-host>_<safe-path>_<unix-ts>_deep.md
Downloaded binaries are saved alongside, under:
{blog_scrape_output_dir}/deep_assets/
Per-post and per-document failures are noted in-band as
`- Fetch error: <code>` / `- Error: <code>` bullets, not by
aborting the whole run. The returned summary dict only carries
an `error` field when discovery itself failed (no posts).
Args:
blog_url: Root URL of the blog (e.g. "https://blog.example.com/").
Returns:
Small summary dict — never the file contents:
{blog_url, output_path, assets_dir, post_count, post_count_ok,
documents_found, documents_extracted, post_byte_count,
document_byte_count, duration_s, discovery}
On failure: {blog_url, error: <stable_code>, message}.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| ctx | No | ||
| blog_url | Yes |
Output Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||