mdbrowse

Compile and browse the web in markdown.

mdb is a web → markdown compiler with a terminal browser on top to make it possible to browse the full web from a terminal window.

Humans get a non-noisy way to browse the web in simple markdown, llms get a token-efficient way to browse the web.

How it Works

When you launch mdb in a terminal, mdb launches an actual Chromium/Chrome browser behind your terminal window. The page loads with all the niceties of a full browser, from JavaScript loading, cookies and more - something W3M could never give you. From that browser window mdb extracts the structure and essence of that page (geometry, landmarks, computed styles — never re-parsed HTML strings). The full DOM and page interaction stay inside the browser; but humans get a zen-like no-fluff browsing experience, and agents get a highly token-efficient view of the web. The compiler classifies each page's shape and emits clean, hierarchically correct markdown that can be snapshotted to archive webpages as clean markdown.

mdb includes an archive store, change-watching sensors, an MCP server for agents, speech output ideal for sight-challenged users.

• Full browser feature-set mdb reads your Safari cookies by default so your logged-in and paywalled-to-you pages render as you'd see them. Or use the --private flag to go without.

• No fluff Light DOM re-ordering ensures articles come out as clean prose that drops most ads and pushes all that menubar and sidebard junk down - and works well with feeds (HN, news sites) presenting one linked line per story

• Site-smart. Write site-handling rules as you like - by default thin-mobile sites (Wikipedia, Stack Overflow, Reddit) are captured with a desktop UA; Reddit uses its .json endpoint for a browser-free structured read (old.reddit HTML when unauthenticated); Cloudflare "Just a moment…" challenges are waited out before capture. Per-host rules live in policy.py (+ ~/.mdb/policy.json).

• Deterministic and diffable. Front-matter carries provenance (source, retrieved, mode, shape+confidence, extractor version) and a body content-hash. Same page state → identical body. Pages become versionable.

• Reader's-choice ad policy. Tracker hosts are blocked at the network; first-party ad furniture (reddit promoted posts, AdSense slots) is dropped by per-host policy rules (policy.py) — each rule carries its why, removal is counted in front-matter (policy_killed), user rules merge from ~/.mdb/policy.json, and MDBROWSE_NO_POLICY=1 turns the layer off.

Simple, Zen browsing from a terminal console

Terminal browsers have always been the calm way to read the web, but have been quietly broken for twenty years: lynx and w3m parse HTML, meanwhile most sites are unreadable without a full JavaScript engine.

mdbrowse is a spiritual successor to those tools in a full browser processor, then delivers the text as md format in the terminal. mdbrowse by default uses keyboard-first vim motions, tabs to hop to focusable links, even forms work such as Wikipedia or DuckDuckGo.

Related MCP server: agentfetch

Built for agents; with a toolbox for analysts

Since LLMs charge per token, reading full HTML is highly inefficient - raw HTML costs about 9× mdb's tokens per fact, and most "clean" extractors pay for their cleanliness by dropping the links or flattening the structure.

In the benchmark suite of tools mdb is the only extractor with 100% fact recall AND navigable links AND surviving structure — the suite table below has the details. The MCP server (mdb-mcp) hands that surface to any agent: fetch with provenance, web search, link filtering, pagination slices served from the capture cache.

The same properties make it an exellent open-source intel tool:

• Capture the web as citable 'point in time' snapshots: Same page state → identical bytes, with front-matter provenance (source URL, retrieval time, auth mode, shape + confidence, extractor version) and a body content-hash. Captures are diffable, versionable, quotable evidence.

• Watch pages for change over time: mdb watch keeps versioned snapshots and fires only on real change — classified readings (ok / changed+diff / error+why), never a bare "page fetched".

• Build your searchable web content memory Everything captured lands in a full-text-searchable archive (mdb search, archive_search) that works offline.

• Read failure feedback: Sites that throw up bot-walls and paywalls come back classified as walls with the reason - you can simply key 'O' no a page to open the URL in your rich browser - likely the cookies you save will let mdb through the next time.

Install

brew install giantravens/tap/mdbrowse     # installs mdb and mdb-mcp

mdb drives your installed Google Chrome when present. Without Chrome, give Playwright its own engine once: playwright install chromium.

From source instead:

git clone https://github.com/GiantRavens/mdbrowse
cd mdbrowse
./mdb --version         # first run builds .venv (via uv) and installs Chromium

The project .venv is host-local. Not mandatory either: any Python 3.11+ environment works if you install mdb into it and give that same environment a Chromium.

Optional authenticated backends

mdb always tries its native browser/compiler first. On a covered X or Reddit URL that classifies as a login wall or application shell, the terminal reader offers an E action to confirm a read-only retry through an installed authenticated backend. Nothing is installed or invoked silently.

mdb setup backends                         # detect tools + show installation/security notes
mdb <x-status-url> --backend opencli       # explicitly use logged-in Chrome
mdb <x-status-url> --backend twitter-cli   # explicitly use X's structured API adapter
mdb <url> --allow-external-fallback        # native first; retry only after wall/app classification

OpenCLI and twitter-cli are separate, optional upstream projects. OpenCLI requires its Chrome Browser Bridge extension and local daemon; twitter-cli extracts X cookies and calls unofficial private APIs. See Authenticated and gated sites before enabling either.

Getting Started

This section assumes you are comfortable copying commands into Terminal, but not necessarily comfortable debugging Python, virtual environments, or browser automation.

What you are installing

mdb is a command-line app. You run it from Terminal, and it opens web pages through a real browser engine in the background. It then shows the page as clean, keyboard-friendly markdown in your terminal.

Three pieces are involved:

• The mdbrowse project folder: the source code you are in now.

• A local Python environment: .venv/ inside this folder. It keeps mdb's Python packages separate from the rest of your computer.

• A Playwright Chromium browser: the browser engine mdb uses for page capture. This is separate from Safari, Chrome, and Firefox.

One-time setup

Open Terminal and go to the project folder. If you keep this repository somewhere else, use that folder instead:

cd ~/Documents/mdbrowse

Run mdb once. The first run creates the local Python environment, installs mdb into it, and installs the browser engine mdb needs:

./mdb --version

You will see first-run setup messages like:

mdb: first-run setup
  project: /Users/you/.../mdbrowse
  venv:    /Users/you/.../mdbrowse/.venv
  phases:  create venv -> install mdb -> install Chromium

If that finishes by printing a version number, setup worked. Later runs reuse the same .venv and start normally.

If you prefer to do the same setup by hand:

uv venv
uv pip install -e .
.venv/bin/playwright install chromium

Set MDBROWSE_NO_BOOTSTRAP=1 if you want ./mdb to fail instead of building the .venv automatically.

Using your own Python environment

You do not have to use the project .venv. If you already manage Python environments with another tool, use Python 3.11 or newer, activate your environment, then install mdb and its browser there:

cd ~/Desktop/notebook/code/mdbrowse
uv pip install -e .
python -m playwright install chromium
mdb --version

The important rule is that mdb, the Python packages, and Playwright's Chromium install must belong to the same active environment. If you use your own environment, run mdb ... instead of ./mdb ...; the repo-root ./mdb launcher is designed around this checkout's .venv.

Your first page

Start with a small, reliable page:

./mdb https://example.com --plain --no-pager

You should see a short markdown page. This proves the Python environment, browser engine, network, and compiler are all working.

Now try the interactive reader:

./mdb https://news.ycombinator.com

Useful first keys:

• j and k move down and up.

• Tab moves to the next link or image.

• Enter opens the focused link.

• H goes back.

• Space previews the focused image, or scrolls when no image is focused. Press Space again to close that preview.

• ? opens help.

• q quits.

Mouse wheel scrolling and clicking links also work in most terminals.

Running mdb from anywhere

The safest command is always ./mdb from inside the project folder.

If your shell has ~/bin on PATH, this repository can also be exposed as mdb from any folder:

mkdir -p ~/bin
ln -sf ~/Desktop/notebook/code/mdbrowse/mdb ~/bin/mdb

Open a new Terminal window and test:

mdb --version

If mdb says "command not found", use ./mdb from the project folder until your shell PATH includes ~/bin.

Common things to do

Read a page interactively:

./mdb https://www.wikipedia.org

Print a page without opening the reader:

./mdb https://example.com --plain

Search the web:

./mdb search "most useful open source intel tools"

Save a page as Markdown in Safari's download folder:

./mdb https://example.com --save
./mdb https://example.com --save-to research/example.md

Add a page to mdb's searchable machine archive:

./mdb https://example.com --archive

Watch a page for future changes:

./mdb watch add https://example.com --name example
./mdb watch scan

Download a linked file:

./mdb get https://example.com/file.pdf

Where files go

The project folder contains code. Generated user data goes somewhere more appropriate for your operating system.

Explicitly saved pages and downloaded links go to Safari's configured download folder on macOS (normally ~/Downloads). In the reader, s prompts for the page filename and d prompts for the download directory; press Return to accept the displayed default.

The searchable agent archive and watch history remain application data:

• macOS: ~/Library/Application Support/mdbrowse/archive and ~/Library/Application Support/mdbrowse/watch

• Linux/BSD: ${XDG_DATA_HOME:-~/.local/share}/mdbrowse/archive and ${XDG_DATA_HOME:-~/.local/share}/mdbrowse/watch

• Windows: %LOCALAPPDATA%\mdbrowse\archive and %LOCALAPPDATA%\mdbrowse\watch

MDBROWSE_DOWNLOADS overrides the default folder for both saved pages and downloaded links. --save-to FILE and mdb get --out DIR provide per-command destinations.

Old folders named ~/mdbrowse-archive or ~/mdbrowse-watch are from older defaults. They are safe to move into the new app-data folders, or you can keep using them by setting MDBROWSE_ARCHIVE and MDBROWSE_WATCH_DIR.

Privacy basics

By default, mdb reads Safari cookies on macOS so pages look like they do when you are signed in. That is useful for sites you already have access to, but it also means mdb is browsing as you.

Homebrew installs bash and zsh completions automatically. For a manual installation, generate the completion script from the executable:

mdb completion bash
mdb completion zsh

Use private mode when you do not want Safari cookies sent:

./mdb https://example.com --private

Saved pages and archives are plain Markdown files on your computer. Do not save private pages unless you are comfortable storing their text locally.

If something goes wrong

If uv is missing, install it first. On macOS with Homebrew:

brew install uv

If first-run setup fails, run the setup steps manually from the project folder so you can see exactly which phase failed:

uv venv
uv pip install -e .

If mdb says the browser is missing:

.venv/bin/playwright install chromium

If you are using your own Python environment instead of .venv, run:

python -m playwright install chromium

If a site blocks the background browser, try a visible browser window:

./mdb https://example.com --headed

If a page is acting strangely because of login state, compare normal and private mode:

./mdb https://example.com
./mdb https://example.com --private

If you only want to check whether the installed copy still works:

./mdb --selftest

Use

mdb                                  # Safari start page (bookmarks, reading list)
mdb news.ycombinator.com             # interactive reader (default in a terminal)
mdb <url> --plain                    # non-interactive render (centered; --no-center)
mdb <url> --raw                      # markdown document with front-matter
mdb <url> --save                     # Markdown in Safari's download folder
mdb <url> --save-to research/a.md    # Markdown at an explicit path
mdb <url> --archive                  # searchable mdb application-data archive
mdb <url> --headed                   # visible real-Chrome window; verification walls (wall shape) trust it
mdb <url> --fallback-headed          # retry headed only after an explicit access-denied wall
mdb <url> --backend opencli          # explicit authenticated Chrome backend (optional install)
mdb <url> --backend twitter-cli      # explicit structured X backend (optional install)
mdb <url> --allow-external-fallback  # native first; authenticated retry only after wall/app
mdb setup backends                   # backend status, installation, and security notes
mdb <url> --speak                    # the page talks (macOS say; --voice, MDBROWSE_VOICE)
mdb <url> --speak-out article.aiff   # page as an audio file
mdb search rust atomics              # web search (DuckDuckGo; MDBROWSE_SEARCH_ENGINE/URL overrides)
mdb feed https://xkcd.com/atom.xml   # RSS/Atom as a feed page
mdb get <file-url>                   # authenticated download (~/Downloads)
mdb oracle <url>                     # judge markdown fidelity against a screenshot
mdb <url> --dump bundle|manifest|body  # inspect any compiler stage
mdb --selftest                       # re-emit the fixture corpus, diff vs goldens

Search defaults to DuckDuckGo now that mdb runs a full Playwright browser. Choose another built-in engine with MDBROWSE_SEARCH_ENGINE=mojeek or MDBROWSE_SEARCH_ENGINE=ddg-html, or provide a custom template with MDBROWSE_SEARCH_URL='https://example.com/search?q={q}'.

Data locations

Human-directed page saves and downloads use Safari's configured Downloads folder (or MDBROWSE_DOWNLOADS). The searchable MCP archive and watch stores default to per-user application data:

• macOS: ~/Library/Application Support/mdbrowse/{archive,watch}

• Linux/BSD: ${XDG_DATA_HOME:-~/.local/share}/mdbrowse/{archive,watch}

• Windows: %LOCALAPPDATA%\mdbrowse\{archive,watch}

MDBROWSE_HOME relocates both stores. MDBROWSE_ARCHIVE and MDBROWSE_WATCH_DIR override the archive or watch store individually. Older ~/mdbrowse-archive and ~/mdbrowse-watch folders are not moved automatically; move them into the new paths or set the env vars above if you want to keep using them in place.

Watch sensors — versioned pages that fire on real change

mdb watch add https://example.com/pricing --name pricing
mdb watch scan          # check all; commits changes to a git store
mdb watch diff pricing  # last change as a patch
mdb watch digest        # Claude narrates the week's changes (briefing material)

Store: the app-data watch directory (git; git log -p <name>.md is the page's history). The trigger hashes visible text only — rotating URL tokens never false-fire.

The reader

Vim-style, with a single focus ring over links, images, and forms (browser-like Tab). Two verbs: Enter = go, Space = peek (preview or close the focused image; page-down otherwise). Every keystroke's effect is predictable from what is visibly highlighted.

Search forms are visible affordances, but they do not auto-focus on page load. Press f for the prompt-driven search flow, or Tab into the field when you want typed characters to go there.

Mouse: wheel scrolls, click follows, click 🖼 previews. (tmux: set -g mouse on.)

Agents and speed

• MCP server (mdb-mcp, registered as mdbrowse): fetch_page (markdown + provenance; long pages paginate via start_char, the continuation served from the capture cache), search_web (results as linked lines), page_links (with a pattern regex filter), archive_page (returns the body hash — compare to detect change), archive_search (full text over the archive: a personal web memory), and the watch fleet — watch_add / watch_list / watch_scan (structured readings: ok / changed+diff / error+why) / watch_diff / watch_remove.

• Agent probe suite (tests/agent_probes.py): live regression guards for the actions agents actually perform — docs code fidelity, pipe tables, search, feed digests, link filtering, pagination stitching, hash determinism, fast classified failure.

• Engine daemon: warm Chromium behind ~/.mdb/engine.sock, auto-spawned on first CLI capture, idle-exit after 30 min. Warm fetches run ~0.7–1.0s. mdb daemon start|stop|status|run; MDBROWSE_DAEMON=off disables.

• Browser execution, token-shaped output: agents are not scraping a TUI transcript. They ride the same real browser capture as the reader, but only the classified markdown page, links, forms, provenance, and requested slices cross the MCP boundary.

How it works

1. Capture — Chromium/Chrome via Playwright, Safari cookies unless --private, stealth shim, tracker/image/media blocking, autoplay suppression, content-stability settle, 3s DNS preflight (black-holed names fail fast with the why). walker.js runs inside the page and emits leaf blocks with landmark, kind, inline-markdown, links, and geometry, plus document-level feed and pagination affordances. page.content() is never taken.

2. Classify — a cheap shape manifest (article | feed | page | app with confidence) from bundle signals, before any emission.

3. Emit — per-shape assembly: repeated-unit detection collapses card fragments to one line per item (shared link target + signature periodicity); headings remap to a strict hierarchy; nav/aside/footer demote to link lists; forms stay out of documents (they're affordances — the reader's f uses them from the bundle).

Every stage is inspectable (--dump), every change is measured, across five suite tiers:

The benchmark compares seven approaches (mdb, raw HTML, tag-strip, Chromium innerText, trafilatura, pandoc, Jina reader) against ground-truth fact signals that no contender defines. Headline numbers (2026-07-05): mdb is the only contender with 100% recall AND navigable links AND surviving structure; raw HTML costs ~9× mdb's tokens per fact; the pandoc pipeline emits 2 tokens for all of HN.

History

v1 (a single-file mdbrowse.py: fetch → strip → convert → repair) was retired on 2026-07-04 after the v2 compiler exceeded it on every axis — see CHANGELOG.md and git history. Its best parts (settle heuristic, binarycookies parser, Safari integration, tracker lists) live on inside v2.