Fetch a web page as clean, deterministic markdown with provenance.

Renders the page in a headless browser (JS/SPA content included), classifies its shape (article / feed / page / app), and emits hierarchically clean markdown. The YAML front-matter carries title, source URL, retrieval timestamp, auth mode, shape verdict with confidence, and a content hash of the body — the same page state always produces the same body, so hashes and diffs are meaningful. Data tables come back as markdown pipe tables.

Browses with the user's Safari cookies by default (logged-in pages render as the user sees them); set private=true for an anonymous fetch. Use wait_selector (a CSS selector) only for SPAs that paint late. Feed/listing pages (HN, news fronts) come back as one linked line per story; article pages as clean prose with inline links.

Long pages paginate: on truncation the tail says which start_char fetches the next slice (served from the capture cache, no re-render).

Optional authenticated backends are never used silently. If native capture is gated on a covered URL, the result explains the available choices. Repeat with backend="opencli" or backend="twitter-cli", or set allow_external_fallback=true to permit the preferred installed backend.

Fetch a page from the Wayback Machine (archive.org) instead of live. Recovers pages the live fetch can't get — bot walls, IP-blocked sites, dead links, or content that changed. Returns the archived snapshot as clean markdown, tagged with its capture date. Reach for this when fetch_page returns a wall, or when you want the historical version of a page.

Search the web; results come back as markdown, one linked line per result. Uses the user's configured engine (DuckDuckGo by default; MDBROWSE_SEARCH_ENGINE or MDBROWSE_SEARCH_URL overrides — e.g. Kagi with the user's session, since searches ride Safari cookies like any page). Follow up with fetch_page on the results worth reading.

Download a linked file directly to disk — a PDF, spec sheet, dataset, image, archive, anything. Fetches through the user's session (Safari cookies + a browser UA + optional Referer ride along), so session-gated and hotlink-protected files come through, and names the file from Content-Disposition or the URL.

Use this when a page LINKS a document worth keeping rather than reading inline — e.g. a solution-brief PDF found via fetch_page/page_links. Pass referer as the page the link was on for hotlink-protected hosts. Saves to ~/Downloads by default (override with dest_dir, or the MDBROWSE_DOWNLOADS env var). Returns the saved path, size, and type. If a hostile WAF tarpits the direct fetch, open the page in the reader and press d (that path fetches through the live browser engine).

Download a video (or just its audio) from a page or media URL via yt-dlp — which handles what a plain file download can't: direct mp4/webm, HLS/DASH manifests (.m3u8/.mpd), and YouTube/Vimeo/embedded players resolved from the page URL.

Muxes best video+audio to mp4, or extracts m4a with audio_only=true. Saves to ~/Downloads by default (override with dest_dir or MDBROWSE_DOWNLOADS). Returns the saved path. For gated video, set MDBROWSE_YTDLP_BROWSER (e.g. 'chrome'/'safari') to ride your browser cookies; it retries anonymously if the cookie backend fails. Use this for a /embed a user wants to keep; use download_document for a plain file (PDF, image, dataset).

List the fillable forms on a page — each form's method, action, submit-button label, and fields (name, type, placeholder, label, select options), plus loose search boxes not wrapped in a . The OBSERVE step before submit_form: call this to learn the field names to fill.

Note: site search often needs no form at all — a GET form just puts the query in the URL, so fetch_page("site.com/search?q=...") (with the field's real name) is simpler when the method is GET.

Fill a form on url and return the RESULT page as clean markdown (same pipeline as fetch_page). fields maps each field's name / label / placeholder to a value, e.g. {"q": "wireless headphones"} or {"From": "SFO", "To": "JFK"}. submit is a button's visible text; if omitted, Enter is pressed in the last filled field (the search-box convention). Rides the user's session, so site search, filters, and logged-in forms work.

Discover field names first with page_forms(url). This unlocks site search, faceted browsing, and any GET/POST/JS form. For a simple GET search, fetch_page with the query in the URL is lighter.

Capture a page and FILE it into document-forge — the 'read it, keep it' bridge. The clean markdown (with provenance front-matter: source URL, retrieval time, content hash) lands as a searchable document in the corpus, so a page you read today is recallable later from your own substrate.

corpus_root defaults to $MDBROWSE_CORPUS_ROOT or /mnt/herfjotur/work; the df CLI path is $MDBROWSE_DF or the notebook default. Use after fetch_page when a page is worth KEEPING, not just reading. For a linked file (PDF/dataset) use download_document instead.

List a page's links as [{text, href}, ...] in document order.

Cheaper to reason over than full markdown when deciding where to navigate next. pattern (case-insensitive regex) filters on link text OR href — e.g. pattern="item?id=" for HN comment pages. Shares the capture with fetch_page on the same URL (60s cache), so calling both costs one page render.

Fetch a page and save a timestamped markdown archive with provenance front-matter (to the mdbrowse app-data archive, or $MDBROWSE_ARCHIVE).

Returns {path, title, shape, hash}. The hash covers the body only, so re-archiving an unchanged page yields the same hash — compare hashes to detect real content changes without diffing.

Start watching a URL for real content change. Takes the first snapshot now (git-committed to the watch store); later watch_scan calls fire only when visible text changes — link-token churn never false-fires. Name defaults to a slug of the URL.

List configured watches: name, url, mode, when last checked and last actually changed.

Re-fetch watches and report one reading each: status ok (no real change) / changed (snapshot committed; diff_sample shows what moved) / error (with the why). Empty names scans everything. Changes are detected on visible text only, so a 'changed' reading means a reader would agree the page changed.

A watch's most recent change as a git patch (old lines -, new lines +). Use after watch_scan reports 'changed' to see exactly what moved.

Stop watching a page. Its snapshot history stays in the store's git log.

Search previously archived pages (the personal web memory that archive_page writes to). Term-AND full text; returns {path, title, source, retrieved, score, snippet} best-first. Read a hit's path for the full page as it was when archived.