fetch
Fetch URLs and convert web content to clean markdown for LLM processing, with options for diff tracking, content focusing, and token-efficient truncation.
Instructions
Fetch a URL and convert to clean markdown for LLM consumption.
Content conversion (automatic by Content-Type):
HTML → clean markdown (boilerplate removed, links preserved)
PDF → markdown with headings and table detection (requires pdf feature)
JSON/plain text → passthrough
SPA data auto-extracted (NEXT_DATA, NUXT, APOLLO_STATE, etc.)
Network features:
HTTP/2 multiplexing, HTTP/3 (QUIC) with 0-RTT
TLS 1.3, Brotli/Zstd/Gzip decompression
Realistic browser fingerprints (Chrome/Firefox/Safari)
Browser cookie injection (Brave/Chrome/Firefox/Safari)
Diff mode (diff: true):
Compares current content against the previous snapshot for this URL
Returns only the changed sections (token-efficient for monitoring tasks)
First fetch caches the page; subsequent fetches return semantic diffs
Unchanged content returns a 5-token confirmation instead of full body
Focus mode (focus: query):
Keeps only sections relevant to the query (BM25 scoring)
Replaces dropped sections with '[N sections omitted]' markers
Diff markers are always preserved regardless of relevance
Token budget (max_tokens: N):
Structure-aware truncation preserving headings, code, and tables
Priority: title > code/tables > headings (30% cap) > body > blockquotes
Returns: Markdown-converted body with timing info (or diff when diff: true).
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| body | Yes | ||
| cookies | No | ||
| diff | Yes | When true, return only changed content vs the previous snapshot. On first fetch the page is cached and full content is returned. On subsequent fetches only the semantic diff is returned, saving tokens for monitoring or change-detection workflows. | |
| focus | No | Natural-language query to focus extraction on relevant sections. When set, uses BM25 scoring to keep only the sections most relevant to the query, replacing omitted sections with count markers. Dramatically reduces token count for large documents when you know what you're looking for. | |
| headers | Yes | ||
| max_tokens | No | Maximum token budget for the returned content. When set, performs structure-aware truncation that preserves headings, code blocks, and tables before trimming body text. Uses priority scoring: title/summary first, then code/tables, then headings (capped at 30% of budget), then body text. | |
| session | No | Named session for cookie persistence across calls. When set, nab uses an isolated per-session cookie jar so that `Set-Cookie` response headers from one call are automatically included on the next call with the same session name. Use this to maintain authenticated state across multiple `fetch` calls after a `login`. Session names: 1-64 chars, alphanumeric + hyphens + underscores. Sessions are created implicitly on first use and live for the process lifetime. Absent = stateless global client (no change). | |
| url | Yes |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The fetched URL | |
| status | Yes | HTTP status code | |
| content | Yes | Markdown-converted body content | |
| has_diff | Yes | True when diff mode was requested and content changed since last snapshot | |
| timing_ms | Yes | Round-trip time in milliseconds | |
| content_type | No | Response Content-Type header |
Implementation Reference
- python/src/nab_loader/core.py:66-110 (handler)The 'fetch' method in 'NabLoader' executes the 'nab' CLI tool to fetch URL content and parse the output into a 'NabResult' object.
def fetch(self, url: str) -> NabResult: """Fetch a single URL and return structured result.""" cmd = [ self.binary, "fetch", url, "--format", "json", "--cookies", self.cookies, "--body", ] try: proc = subprocess.run( cmd, capture_output=True, text=True, timeout=self.timeout, ) except FileNotFoundError: raise NabNotFoundError() except subprocess.TimeoutExpired: raise NabFetchError(url, f"timed out after {self.timeout}s") if proc.returncode != 0: raise NabFetchError(url, proc.stderr.strip() or f"exit code {proc.returncode}") # nab --format json outputs JSON on first line, body follows lines = proc.stdout.split("\n", 1) try: meta = json.loads(lines[0]) except (json.JSONDecodeError, IndexError): raise NabFetchError(url, "could not parse nab JSON output") markdown = lines[1] if len(lines) > 1 else "" return NabResult( url=meta.get("url", url), markdown=markdown, status=meta.get("status", 0), size=meta.get("size", len(markdown)), time_ms=meta.get("time_ms", 0.0), metadata=meta, )