pulldown

Pull down web pages as clean Markdown for LLM agents.

• HTTP-first with browser-like defaults

• Optional Chromium rendering for JS-heavy pages

• Five detail levels: minimal, readable, structured, full, raw

• Core installs decode Brotli-compressed pages correctly

• Page-type aware routing with nested meta["routing"] diagnostics

• Concurrent batch fetching with fetch_many()

• Bounded site crawling with robots.txt support and per-domain politeness

• Validator-based caching (ETag / Last-Modified) with atomic writes

• SSRF guards: private/loopback/metadata addresses blocked by default

• Response size caps and transient-error retries

• CLI, Python API, and MCP server

Install

pip install pulldown                 # core
pip install 'pulldown[render]'       # + Playwright (Chromium rendering)
pip install 'pulldown[mcp]'          # + MCP server
pip install 'pulldown[all]'          # everything

Core installs include Brotli support, so br-compressed HTML is decoded before minimal, readable, full, or raw processing.

Core installs also include lxml_html_clean, avoiding the missing-helper import issue some agent sandboxes hit on older releases.

For rendered pages, also run playwright install chromium once.

Related MCP server: WebforAI Text Extractor

Quick Start

CLI

pulldown get https://example.com
pulldown get https://example.com --detail minimal
pulldown get https://example.com --detail structured
pulldown get https://example.com --render --scroll 3
pulldown crawl https://docs.example.com --max-pages 20 --delay-ms 200
pulldown bench https://example.com --runs 5
pulldown cache stats

Python

import asyncio
from pulldown import fetch, fetch_many, crawl, Detail, PageCache

async def main():
    # Single fetch
    result = await fetch("https://example.com", detail=Detail.readable)
    print(result.title)
    print(result.meta["routing"])

    # Batch fetch with caching
    cache = PageCache(ttl=3600)
    results = await fetch_many(
        ["https://a.com", "https://b.com"],
        concurrency=5,
        cache=cache,
        retries=2,
    )

    # Crawl a docs site
    crawl_result = await crawl(
        "https://docs.example.com/",
        max_pages=50,
        max_depth=2,
        respect_robots=True,
        per_domain_delay_ms=200,
    )
    markdown = crawl_result.to_markdown()

asyncio.run(main())

MCP

Add to your client config (e.g. Claude Desktop):

{
  "mcpServers": {
    "pulldown": {
      "command": "python",
      "args": ["-m", "pulldown.mcp_server"],
      "env": {
        "PULLDOWN_CACHE_DIR": "~/.cache/pulldown"
      }
    }
  }
}

Environment variables:

Detail Levels

readable now routes dashboard and listing pages toward a structured extractor instead of trying to flatten them into pseudo-articles. Result metadata includes the detected page type and extraction strategy under meta["routing"] so agents can branch explicitly.

Example routing payload:

{
    "page_type": "listing",
    "source": "rules",
    "confidence": 1.0,
    "abstained": False,
    "strategy_used": "structured",
    "quality_grade": "high",
    "render_recommended": False,
}

Use --routing-log path.jsonl in the CLI or routing_log_path="path.jsonl" in Python to capture feature vectors, probabilities, fallback decisions, and quality outcomes for offline retraining.

Security

pulldown refuses to fetch URLs that resolve to private, loopback, link-local, or cloud-metadata addresses by default. This prevents LLM-driven SSRF into internal services (e.g., AWS metadata at 169.254.169.254, Redis on localhost:6379). Override with allow_private_addresses=True if you understand the risk.

Responses above 10 MiB are rejected by default (max_bytes parameter).

Only http and https schemes are accepted; file:, ftp:, etc. are rejected.

MCP Metadata

The MCP tools keep their default plain-content behavior, but callers can ask for structured metadata explicitly:

await pulldown("https://example.com", include_meta=True)

That JSON response includes the same nested meta["routing"] object returned by the Python API.

License

MIT