fetch

fetch

Fetch URLs and convert web content to clean markdown for LLM processing, with options for diff tracking, content focusing, and token-efficient truncation.

Instructions

Fetch a URL and convert to clean markdown for LLM consumption.

Content conversion (automatic by Content-Type):

HTML → clean markdown (boilerplate removed, links preserved)
PDF → markdown with headings and table detection (requires pdf feature)
JSON/plain text → passthrough
SPA data auto-extracted (NEXT_DATA, NUXT, APOLLO_STATE, etc.)

Network features:

HTTP/2 multiplexing, HTTP/3 (QUIC) with 0-RTT
TLS 1.3, Brotli/Zstd/Gzip decompression
Realistic browser fingerprints (Chrome/Firefox/Safari)
Browser cookie injection (Brave/Chrome/Firefox/Safari)

Diff mode (diff: true):

Compares current content against the previous snapshot for this URL
Returns only the changed sections (token-efficient for monitoring tasks)
First fetch caches the page; subsequent fetches return semantic diffs
Unchanged content returns a 5-token confirmation instead of full body

Focus mode (focus: query):

Keeps only sections relevant to the query (BM25 scoring)
Replaces dropped sections with '[N sections omitted]' markers
Diff markers are always preserved regardless of relevance

Token budget (max_tokens: N):

Structure-aware truncation preserving headings, code, and tables
Priority: title > code/tables > headings (30% cap) > body > blockquotes

Returns: Markdown-converted body with timing info (or diff when diff: true).

Input Schema

TableJSON Schema

Name	Required	Description
`body`	Yes
`cookies`	No
`diff`	Yes	When true, return only changed content vs the previous snapshot. On first fetch the page is cached and full content is returned. On subsequent fetches only the semantic diff is returned, saving tokens for monitoring or change-detection workflows.
`focus`	No	Natural-language query to focus extraction on relevant sections. When set, uses BM25 scoring to keep only the sections most relevant to the query, replacing omitted sections with count markers. Dramatically reduces token count for large documents when you know what you're looking for.
`headers`	Yes
`max_tokens`	No	Maximum token budget for the returned content. When set, performs structure-aware truncation that preserves headings, code blocks, and tables before trimming body text. Uses priority scoring: title/summary first, then code/tables, then headings (capped at 30% of budget), then body text.
`session`	No	Named session for cookie persistence across calls. When set, nab uses an isolated per-session cookie jar so that `Set-Cookie` response headers from one call are automatically included on the next call with the same session name. Use this to maintain authenticated state across multiple `fetch` calls after a `login`. Session names: 1-64 chars, alphanumeric + hyphens + underscores. Sessions are created implicitly on first use and live for the process lifetime. Absent = stateless global client (no change).
`url`	Yes

Implementation Reference

python/src/nab_loader/core.py:66-110 (handler)

The 'fetch' method in 'NabLoader' executes the 'nab' CLI tool to fetch URL content and parse the output into a 'NabResult' object.

def fetch(self, url: str) -> NabResult:
    """Fetch a single URL and return structured result."""
    cmd = [
        self.binary,
        "fetch",
        url,
        "--format",
        "json",
        "--cookies",
        self.cookies,
        "--body",
    ]
    try:
        proc = subprocess.run(
            cmd,
            capture_output=True,
            text=True,
            timeout=self.timeout,
        )
    except FileNotFoundError:
        raise NabNotFoundError()
    except subprocess.TimeoutExpired:
        raise NabFetchError(url, f"timed out after {self.timeout}s")

    if proc.returncode != 0:
        raise NabFetchError(url, proc.stderr.strip() or f"exit code {proc.returncode}")

    # nab --format json outputs JSON on first line, body follows
    lines = proc.stdout.split("\n", 1)
    try:
        meta = json.loads(lines[0])
    except (json.JSONDecodeError, IndexError):
        raise NabFetchError(url, "could not parse nab JSON output")

    markdown = lines[1] if len(lines) > 1 else ""

    return NabResult(
        url=meta.get("url", url),
        markdown=markdown,
        status=meta.get("status", 0),
        size=meta.get("size", len(markdown)),
        time_ms=meta.get("time_ms", 0.0),
        metadata=meta,
    )

nab

Instructions

Input Schema

Implementation Reference

Other Tools

Latest Blog Posts

MCP directory API