# pulldown

Pull down web pages as clean Markdown for LLM agents.

- HTTP-first with browser-like defaults
- Optional Chromium rendering for JS-heavy pages
- Five detail levels: `minimal`, `readable`, `structured`, `full`, `raw`
- Core installs decode Brotli-compressed pages correctly
- Page-type aware routing with nested `meta["routing"]` diagnostics
- Concurrent batch fetching with `fetch_many()`
- Bounded site crawling with `robots.txt` support and per-domain politeness
- Validator-based caching (ETag / Last-Modified) with atomic writes
- SSRF guards: private/loopback/metadata addresses blocked by default
- Response size caps and transient-error retries
- CLI, Python API, and MCP server

## Install

```bash
pip install pulldown                 # core
pip install 'pulldown[render]'       # + Playwright (Chromium rendering)
pip install 'pulldown[mcp]'          # + MCP server
pip install 'pulldown[all]'          # everything
```

Core installs include Brotli support, so `br`-compressed HTML is decoded before
`minimal`, `readable`, `full`, or `raw` processing.

Core installs also include `lxml_html_clean`, avoiding the missing-helper import
issue some agent sandboxes hit on older releases.

For rendered pages, also run `playwright install chromium` once.

## Quick Start

### CLI

```bash
pulldown get https://example.com
pulldown get https://example.com --detail minimal
pulldown get https://example.com --detail structured
pulldown get https://example.com --render --scroll 3
pulldown crawl https://docs.example.com --max-pages 20 --delay-ms 200
pulldown bench https://example.com --runs 5
pulldown cache stats
```

### Python

```python
import asyncio
from pulldown import fetch, fetch_many, crawl, Detail, PageCache

async def main():
    # Single fetch
    result = await fetch("https://example.com", detail=Detail.readable)
    print(result.title)
    print(result.meta["routing"])

    # Batch fetch with caching
    cache = PageCache(ttl=3600)
    results = await fetch_many(
        ["https://a.com", "https://b.com"],
        concurrency=5,
        cache=cache,
        retries=2,
    )

    # Crawl a docs site
    crawl_result = await crawl(
        "https://docs.example.com/",
        max_pages=50,
        max_depth=2,
        respect_robots=True,
        per_domain_delay_ms=200,
    )
    markdown = crawl_result.to_markdown()

asyncio.run(main())
```

### MCP

Add to your client config (e.g. Claude Desktop):

```json
{
  "mcpServers": {
    "pulldown": {
      "command": "python",
      "args": ["-m", "pulldown.mcp_server"],
      "env": {
        "PULLDOWN_CACHE_DIR": "~/.cache/pulldown"
      }
    }
  }
}
```

Environment variables:

| Variable | Default | Purpose |
|---|---|---|
| `MCP_TRANSPORT` | `stdio` | `stdio` or `http` |
| `MCP_HOST` | `127.0.0.1` | Bind address for HTTP transport |
| `MCP_PORT` | `8080` | Port for HTTP transport |
| `PULLDOWN_CACHE_DIR` | _unset_ | Enable caching to this directory |
| `PULLDOWN_CACHE_TTL` | `3600` | Cache TTL in seconds |
| `PULLDOWN_ALLOW_PRIVATE` | `0` | Set to `1` to allow private addresses |
| `PULLDOWN_ROUTING_LOG` | _unset_ | Append per-page routing diagnostics JSONL |

## Detail Levels

| Level | Output | Best for |
|---|---|---|
| `minimal` | Title + plain text | Lowest-token summarisation |
| `readable` | Auto-routed readable Markdown with links | Default. Uses article extraction for narrative pages and routes non-article pages to a better strategy |
| `structured` | Hierarchy-preserving Markdown with summarized tables | Dashboards, listings, landing pages, and table-heavy app views |
| `full` | Full-page Markdown incl. chrome | Pages without clear article body |
| `raw` | Untouched HTML | Custom parsing downstream |

`readable` now routes dashboard and listing pages toward a structured extractor
instead of trying to flatten them into pseudo-articles. Result metadata includes
the detected page type and extraction strategy under `meta["routing"]` so
agents can branch explicitly.

Example routing payload:

```python
{
    "page_type": "listing",
    "source": "rules",
    "confidence": 1.0,
    "abstained": False,
    "strategy_used": "structured",
    "quality_grade": "high",
    "render_recommended": False,
}
```

Use `--routing-log path.jsonl` in the CLI or `routing_log_path="path.jsonl"`
in Python to capture feature vectors, probabilities, fallback decisions, and
quality outcomes for offline retraining.

## Security

pulldown refuses to fetch URLs that resolve to private, loopback,
link-local, or cloud-metadata addresses by default. This prevents
LLM-driven SSRF into internal services (e.g., AWS metadata at
`169.254.169.254`, Redis on `localhost:6379`). Override with
`allow_private_addresses=True` if you understand the risk.

Responses above 10 MiB are rejected by default (`max_bytes` parameter).

Only `http` and `https` schemes are accepted; `file:`, `ftp:`, etc. are
rejected.

## MCP Metadata

The MCP tools keep their default plain-content behavior, but callers can ask
for structured metadata explicitly:

```python
await pulldown("https://example.com", include_meta=True)
```

That JSON response includes the same nested `meta["routing"]` object returned
by the Python API.

## License

MIT