# escalator

Give it a URL, get clean Markdown. It climbs the cheapest rung that works — a
plain HTTP fetch, the same fetch through a residential proxy, then a stealth
browser — and stops at the first one that comes back with real content.

```
$ escalator scrape https://en.wikipedia.org/wiki/Web_scraping | head -3
# Web scraping

**Web scraping**, **web harvesting**, or **web data extraction** is [data scraping](...)
```

This README is an **installation manual**. It walks the whole way from a bare
machine to a running, authenticated HTTP API on three targets — Linux, macOS,
and Docker — with nothing left to guess. If you only want the one-page tour of
*why* it is shaped this way, that lives in [DESIGN.md](DESIGN.md).

## Contents

- [What you are installing](#what-you-are-installing)
- [Install on Linux](#install-on-linux)
  - [1. System libraries the browser needs](#1-system-libraries-the-browser-needs)
  - [2. Install Python and escalator](#2-install-python-and-escalator)
  - [3. Give it a browser](#3-give-it-a-browser)
  - [4. Configure it](#4-configure-it)
  - [5. Check the install](#5-check-the-install)
  - [6. Run it as an API](#6-run-it-as-an-api-linux)
- [Install on macOS](#install-on-macos)
- [Install with Docker](#install-with-docker)
- [Configuration reference](#configuration-reference)
- [Using the API](#using-the-api)
- [Command reference](#command-reference)
- [The ladder](#the-ladder)
- [Troubleshooting](#troubleshooting)
- [Development](#development) · [License](#license)

## What you are installing

`escalator` is a single Python package (`escalator` on PyPI) that is three things
in one binary:

- a **CLI** — `escalator scrape URL` prints Markdown to stdout, so it pipes;
- an **HTTP API** — `escalator serve` exposes `POST /scrape`, bearer-authenticated;
- an **MCP endpoint** — the same ladder as one tool at `/mcp`, for agents (the
  `escalator[mcp]` extra).

The pieces you need to have present:

| Requirement | Why | Notes |
|---|---|---|
| **Python 3.11–3.13** | The runtime. | Not 3.14 yet — the browser driver won't import there. `uv` will fetch a 3.13 for you if you don't have one. |
| **A Chrome-family browser** | The top rung renders JS and defeats Cloudflare-class walls. | escalator brings none with it. It uses one already on the machine, or downloads its own with `escalator browser install`. |
| **Shared libraries (headless Linux only)** | Chrome links against ~20 system `.so` files a desktop already has but a slim server does not. | Listed below per distro. macOS and desktop Linux already have them. |
| **A residential proxy** *(optional)* | The middle rungs; needed only for sites that ban datacenter IPs. | Everything works without one — the two proxied rungs simply skip themselves. |

It writes nothing outside its own config and data directories, ships no
telemetry, and bundles no browser or proxies.

---

## Install on Linux

The instructions below are distro-neutral. Pick the package-manager line that
matches yours; everything after step 1 is identical.

### 1. System libraries the browser needs

On a desktop these are already present — skip to step 2. On a **slim server or
container** Chrome will resolve but refuse to launch with
`error while loading shared libraries: libnss3.so` until you install them.

**Debian / Ubuntu** (apt):

```bash
sudo apt-get update && sudo apt-get install -y \
  libnss3 libnspr4 libatk1.0-0t64 libatk-bridge2.0-0t64 libcups2t64 libdrm2 \
  libxkbcommon0 libxcomposite1 libxdamage1 libxext6 libxfixes3 libxrandr2 \
  libgbm1 libglib2.0-0t64 libpango-1.0-0 libcairo2 libasound2t64 \
  libatspi2.0-0t64 libxcb1 libdbus-1-3 libexpat1
```

> The `t64` suffixes are Ubuntu 24.04 / Debian 13. On older releases apt will
> tell you the names it wants (`libatk1.0-0`, `libcups2`, `libasound2`, …) —
> drop the `t64` and re-run.

**Fedora / RHEL / CentOS Stream / Rocky / Alma** (dnf):

```bash
sudo dnf install -y \
  nss nspr atk at-spi2-atk at-spi2-core cups-libs libdrm \
  libxkbcommon libXcomposite libXdamage libXext libXfixes libXrandr \
  mesa-libgbm glib2 pango cairo alsa-lib libxcb dbus-libs expat
```

**Arch / Manjaro** (pacman):

```bash
sudo pacman -S --needed \
  nss nspr atk at-spi2-atk at-spi2-core cups libdrm \
  libxkbcommon libxcomposite libxdamage libxext libxfixes libxrandr \
  mesa glib2 pango cairo alsa-lib libxcb dbus expat
```

**Alpine** (musl libc): Chrome for Testing is a glibc build and will not run
here. Install the distro's chromium instead and point escalator at it:

```bash
sudo apk add chromium nss freetype harfbuzz ttf-freefont
export ESCALATOR_BROWSER_PATH=/usr/bin/chromium-browser
```

> **Shortcut for any distro:** installing your distro's own `chromium` (or
> `google-chrome`) package pulls in *every* one of these libraries as a
> dependency. You can install it purely to satisfy the libs and still let
> escalator drive its own pinned Chrome — or just use that browser directly.

### 2. Install Python and escalator

Use **[uv](https://docs.astral.sh/uv/)** — it installs the tool onto your PATH
and provisions a correct Python (3.11–3.13) even if the system one is too new or
too old.

```bash
# install uv itself (once)
curl -LsSf https://astral.sh/uv/install.sh | sh
exec $SHELL              # pick up the new PATH

# install escalator, with the MCP endpoint included
uv tool install "escalator[mcp]"
```

That puts an `escalator` binary in `~/.local/bin`. Confirm PATH is set:

```bash
escalator --version
# escalator 0.1.0 | python 3.13.x | Linux-x86_64
```

<details>
<summary>Prefer pipx, or a plain venv?</summary>

```bash
# pipx — same idea as uv tool
pipx install "escalator[mcp]"

# or an explicit virtualenv you manage yourself
python3 -m venv ~/.venvs/escalator
~/.venvs/escalator/bin/pip install "escalator[mcp]"
~/.venvs/escalator/bin/escalator --version
```

Drop the `[mcp]` extra if you don't need the `/mcp` endpoint. It is the only
optional dependency; the scraping ladder, the browser rung, and the REST API are
all in the base install.
</details>

### 3. Give it a browser

escalator never downloads software as a side effect of a scrape. Provision the
browser once, explicitly. Either is fine:

```bash
# A) download a pinned Chrome for Testing into escalator's own data dir
escalator browser install

# B) or just rely on a Chrome/Chromium/Edge/Brave already installed — check:
escalator browser list      # shows every candidate and marks the winner (→)
```

### 4. Configure it

Two supported styles — use whichever fits how you'll run it:

**Interactive**, writes `~/.config/escalator/config.toml`:

```bash
escalator init              # add --yes to take every default non-interactively
```

**Environment only** (no file) — this is how the API service below is
configured. Every setting has an `ESCALATOR_*` variable; see the
[configuration reference](#configuration-reference).

### 5. Check the install

```bash
escalator doctor
```

`doctor` checks Python, the config file, the data directory, browser
resolution, an **actual headless launch** (this is what catches missing libs),
and — if a proxy is set — one real request through it, reporting the egress IP
and country with the password masked. Every ❌ prints the one line that fixes it,
and the exit code is non-zero on any failure, so scripts can gate on it. Add
`--json` for machines.

You are done with the CLI. To expose it as a network service, continue.

### 6. Run it as an API (Linux)

The goal: escalator running as an unprivileged system service, bound to
loopback, with an nginx reverse proxy terminating TLS in front of it.

#### 6a. Create a dedicated user, venv, and data directory

Isolating the service from your login account keeps its data, browser, and
crash blast-radius contained.

```bash
# a locked-down system user that owns the install and its data
sudo useradd --system --home-dir /opt/escalator --shell /usr/sbin/nologin escalator
sudo install -d -o escalator -g escalator /opt/escalator /var/lib/escalator

# install escalator into a venv owned by that user
sudo -u escalator python3 -m venv /opt/escalator/venv
sudo -u escalator /opt/escalator/venv/bin/pip install --upgrade pip
sudo -u escalator /opt/escalator/venv/bin/pip install "escalator[mcp]"

# give the service its own browser, inside its data dir
sudo -u escalator env ESCALATOR_STORAGE_DATA_DIR=/var/lib/escalator \
  /opt/escalator/venv/bin/escalator browser install
```

#### 6b. Mint an API key and write the environment file

The API rejects every request unless at least one bearer key is configured. The
`server.api_keys` list *is* the source of truth — removing a key revokes it on
the next restart. Mint one with the built-in command; `--show` prints a fresh key
to stdout without writing any file, which is exactly what an env-based deploy
wants:

```bash
KEY=$(sudo -u escalator /opt/escalator/venv/bin/escalator key generate --show)
echo "$KEY"     # save this — it is the credential
```

Put configuration in an env file the service reads (root-owned, mode 600 — it
holds the key and any proxy password):

```bash
sudo install -d -m 755 /etc/escalator
sudo tee /etc/escalator/escalator.env >/dev/null <<EOF
# --- required ---
ESCALATOR_SERVER_API_KEYS=$KEY
# Bind to loopback: nginx (below) is the only thing that should reach it.
ESCALATOR_SERVER_HOST=127.0.0.1
ESCALATOR_SERVER_PORT=8000
ESCALATOR_STORAGE_DATA_DIR=/var/lib/escalator

# --- optional: a residential proxy for the middle rungs ---
# ESCALATOR_PROXY_ENABLED=true
# ESCALATOR_PROXY_URL=http://user:pass@proxy.example:8080
# several exits, round-robin:
# ESCALATOR_PROXY_LIST=http://user:pass@a.example:8080,http://user:pass@b.example:8080

# --- optional: politeness ---
ESCALATOR_POLITENESS_RESPECT_ROBOTS=true
ESCALATOR_POLITENESS_RATE_LIMIT_RPS=1.0
EOF
sudo chmod 600 /etc/escalator/escalator.env
```

Multiple keys are comma-separated: `ESCALATOR_SERVER_API_KEYS=key1,key2`.

#### 6c. Supervise the process

Use **systemd** (native to every modern Linux) or **Supervisor** — pick one.

**systemd** — write `/etc/systemd/system/escalator.service`:

```ini
[Unit]
Description=escalator scraping gateway
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=escalator
Group=escalator
EnvironmentFile=/etc/escalator/escalator.env
ExecStart=/opt/escalator/venv/bin/escalator serve
Restart=on-failure
RestartSec=3

# hardening — the service only needs to write its own data dir
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/escalator

[Install]
WantedBy=multi-user.target
```

```bash
sudo systemctl daemon-reload
sudo systemctl enable --now escalator
sudo systemctl status escalator
journalctl -u escalator -f          # follow the logs
curl -s localhost:8000/healthz      # {"status":"ok","version":"0.1.0"}
```

**Supervisor** — as an alternative, drop this into
`/etc/supervisor/conf.d/escalator.conf`:

```ini
[program:escalator]
command=/opt/escalator/venv/bin/escalator serve
user=escalator
directory=/var/lib/escalator
autostart=true
autorestart=true
startsecs=5
stopasgroup=true
killasgroup=true
; Supervisor splits `environment` on commas, so a value that itself contains
; commas (several API keys, a proxy list) MUST be quoted as one item:
environment=ESCALATOR_SERVER_API_KEYS="PASTE_KEY_HERE",ESCALATOR_SERVER_HOST="127.0.0.1",ESCALATOR_SERVER_PORT="8000",ESCALATOR_STORAGE_DATA_DIR="/var/lib/escalator"
stdout_logfile=/var/log/escalator.log
redirect_stderr=true
```

```bash
sudo supervisorctl reread && sudo supervisorctl update
sudo supervisorctl status escalator
```

#### 6d. Put a web server in front of it (TLS)

escalator serves plain HTTP on loopback and runs as a single process — it is not
meant to face the internet directly. Terminate TLS and expose it with nginx.
Write `/etc/nginx/sites-available/escalator`:

```nginx
server {
    listen 80;
    server_name scrape.example.com;
    # redirect everything to HTTPS (certbot will fill in the 443 block)
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    http2 on;
    server_name scrape.example.com;

    ssl_certificate     /etc/letsencrypt/live/scrape.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/scrape.example.com/privkey.pem;

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # A browser climb can take 30s+. Don't let nginx hang up mid-scrape.
        proxy_read_timeout 120s;
        proxy_connect_timeout 10s;
    }
}
```

```bash
sudo ln -s /etc/nginx/sites-available/escalator /etc/nginx/sites-enabled/
sudo nginx -t                                   # check config
sudo certbot --nginx -d scrape.example.com      # obtain + wire up a cert
sudo systemctl reload nginx
```

Test end to end:

```bash
curl -s https://scrape.example.com/healthz
curl -s https://scrape.example.com/scrape \
  -H "Authorization: Bearer PASTE_KEY_HERE" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}' | head
```

Prefer [Caddy](https://caddyserver.com/)? A whole `Caddyfile` for automatic TLS
is just:

```caddy
scrape.example.com {
    reverse_proxy 127.0.0.1:8000
}
```

---

## Install on macOS

For desktop use and local development. The system libraries step does not apply —
macOS already has what the browser needs.

### 1. Prerequisites

```bash
# Homebrew, if you don't have it: https://brew.sh
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

# uv, to install the tool and manage its Python
brew install uv
```

A Chrome-family browser is usually already installed. If not, either
`brew install --cask google-chrome`, or let escalator fetch its own in step 3.

### 2. Install escalator

```bash
uv tool install "escalator[mcp]"
escalator --version
# escalator 0.1.0 | python 3.13.x | Darwin-arm64
```

### 3. Browser, config, and check

```bash
escalator browser list       # is Chrome/Chromium/Edge/Brave already here?
escalator browser install    # …or download a pinned one into the data dir

escalator init               # write ~/Library/Application Support/escalator/config.toml
escalator doctor             # verify everything, including a real headless launch
```

You can now `escalator scrape https://example.com`. To run the API continuously
on the Mac, continue.

### 4. Run it as an API (macOS)

#### 4a. Configuration

Mint a key straight into the `init` config file — no hand-editing:

```bash
escalator key generate
# → prints the key and appends it to ~/Library/Application Support/escalator/config.toml
```

That writes `api_keys = ["…"]` under `[server]`. While you're there you can set
`host` and `port` too, though `127.0.0.1:8000` is already the default.

(Or skip the file and pass everything as `ESCALATOR_*` env vars in the plist
below — the plist wins over the file either way. `escalator key generate --show`
prints a fresh key for that without touching the config.)

#### 4b. Keep it running with launchd

launchd is the native macOS supervisor. Create
`~/Library/LaunchAgents/com.escalator.serve.plist`:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
  "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.escalator.serve</string>

    <key>ProgramArguments</key>
    <array>
        <string>/Users/YOU/.local/bin/escalator</string>
        <string>serve</string>
    </array>

    <key>EnvironmentVariables</key>
    <dict>
        <key>ESCALATOR_SERVER_API_KEYS</key>
        <string>PASTE_A_GENERATED_KEY</string>
        <key>ESCALATOR_SERVER_HOST</key>
        <string>127.0.0.1</string>
        <key>ESCALATOR_SERVER_PORT</key>
        <string>8000</string>
    </dict>

    <key>RunAtLoad</key>   <true/>
    <key>KeepAlive</key>   <true/>
    <key>StandardOutPath</key>  <string>/tmp/escalator.log</string>
    <key>StandardErrorPath</key><string>/tmp/escalator.err</string>
</dict>
</plist>
```

Replace `/Users/YOU` with your home (`echo $HOME`) and confirm the binary path
with `which escalator`. Then:

```bash
launchctl load ~/Library/LaunchAgents/com.escalator.serve.plist
curl -s localhost:8000/healthz
# to stop: launchctl unload ~/Library/LaunchAgents/com.escalator.serve.plist
```

#### 4c. A web server in front (optional)

If this Mac serves other machines, put nginx in front for TLS exactly as in the
[Linux nginx step](#6d-put-a-web-server-in-front-of-it-tls):

```bash
brew install nginx        # config lives at /opt/homebrew/etc/nginx/
```

Use the same `server { … }` block; on a Mac you'll typically manage certs with
`brew install certbot` or point at certs you already have.

---

## Install with Docker

The published image (`ghcr.io/ruslanstarikov/escalator`) carries a pinned Chrome
and the MCP extra, runs as an unprivileged user, and is configured **entirely by
environment variable** — no config file is ever mounted. This is the least
fiddly way to run the API on a server.

### One-shot commands

The default command prints help; give it a subcommand to do work.

```bash
# check the image on this host (browser launch, libs, everything)
docker run --rm ghcr.io/ruslanstarikov/escalator doctor

# scrape a single URL to stdout
docker run --rm ghcr.io/ruslanstarikov/escalator scrape https://example.com

# mint a bearer key to paste into the env below (--show writes no file)
docker run --rm ghcr.io/ruslanstarikov/escalator key generate --show
```

### Run the API with docker run

```bash
docker run -d --name escalator \
  -e ESCALATOR_SERVER_API_KEYS=PASTE_A_GENERATED_KEY \
  -p 127.0.0.1:8000:8000 \
  -v escalator-data:/data \
  --shm-size=512m \
  --restart unless-stopped \
  ghcr.io/ruslanstarikov/escalator serve
```

What each flag is doing, so nothing is mysterious:

| Flag | Why it's there |
|---|---|
| `-e ESCALATOR_SERVER_API_KEYS=…` | **Required.** Without a key every request is rejected. Comma-separate several. |
| `-p 127.0.0.1:8000:8000` | Publish to loopback only; a reverse proxy (below) faces the network. Drop the `127.0.0.1:` to expose it directly. |
| `-v escalator-data:/data` | Persist the SQLite DB (keys cache, tier cache, request log) and the browser across restarts. |
| `--shm-size=512m` | Chromium needs more than Docker's default 64 MB of shared memory or it crashes on heavy pages. |
| `--restart unless-stopped` | Bring it back after a reboot or a crash. |

The image already sets `ESCALATOR_SERVER_HOST=0.0.0.0`,
`ESCALATOR_STORAGE_DATA_DIR=/data`, and `ESCALATOR_BROWSER_PATH` to the baked-in
browser, so you don't repeat those.

```bash
docker logs -f escalator
curl -s localhost:8000/healthz
```

### Run the API with Docker Compose (recommended)

Create a `.env` next to the compose file (never commit it):

```dotenv
ESCALATOR_SERVER_API_KEYS=PASTE_A_GENERATED_KEY
# optional residential proxy:
# ESCALATOR_PROXY_ENABLED=true
# ESCALATOR_PROXY_URL=http://user:pass@proxy.example:8080
```

`docker-compose.yml`:

```yaml
services:
  escalator:
    image: ghcr.io/ruslanstarikov/escalator:latest
    command: ["serve"]
    environment:
      ESCALATOR_SERVER_API_KEYS: "${ESCALATOR_SERVER_API_KEYS:?set a key in .env}"
      ESCALATOR_PROXY_ENABLED:   "${ESCALATOR_PROXY_ENABLED:-false}"
      ESCALATOR_PROXY_URL:       "${ESCALATOR_PROXY_URL:-}"
      ESCALATOR_BROWSER_VIA_PROXY: "true"
      ESCALATOR_POLITENESS_RESPECT_ROBOTS: "true"
      ESCALATOR_POLITENESS_RATE_LIMIT_RPS: "1.0"
    ports:
      - "127.0.0.1:8000:8000"   # loopback; put a reverse proxy in front for TLS
    volumes:
      - ./data:/data            # must be writable by uid 1000 (see note)
    restart: unless-stopped
    shm_size: 512mb
```

```bash
docker compose up -d
docker compose logs -f
curl -s localhost:8000/healthz
```

A ready-to-copy version with the full proxy wiring is in
[`docker-compose.example.yml`](docker-compose.example.yml).

> **Volume permissions.** The container runs as **uid 1000**. Docker Desktop
> (macOS/Windows) maps this automatically. On a Linux host, a bind-mounted
> `./data` must be writable by that uid — either `sudo chown -R 1000:1000 ./data`
> once, or add `user: "$(id -u):$(id -g)"` to the service. A named volume
> (`escalator-data:/data`) sidesteps this entirely.

### TLS in front of the container

Terminate TLS with the same nginx block from the
[Linux section](#6d-put-a-web-server-in-front-of-it-tls) pointing at
`http://127.0.0.1:8000`, or add a Caddy/Traefik service to the compose file.
Because the container publishes only to `127.0.0.1`, the proxy is the sole path
in from the network.

### Build the image yourself

```bash
git clone https://github.com/ruslanstarikov/escalator.git
cd escalator
docker build -t escalator:local .
docker run --rm escalator:local doctor
```

The build pins its Chrome version via a `--build-arg CHROME_VERSION=…`; see the
comments at the top of the [`Dockerfile`](Dockerfile). On arm64 (where Chrome for
Testing has no build) it installs the distro's chromium instead, so the same
`ESCALATOR_BROWSER_PATH` works on both architectures.

---

## Configuration reference

One file, written by `init`, at the platform config dir
(`~/.config/escalator/config.toml` on Linux, `~/Library/Application
Support/escalator/config.toml` on macOS). Override the location with `--config`.

Precedence, everywhere:

```
CLI flag  >  environment  >  config.toml  >  default
```

Every key has an environment variable — which is how the Docker image and the
system service are configured without a file at all:

| config key | env var | default | what it does |
|---|---|---|---|
| `browser.path` | `ESCALATOR_BROWSER_PATH` | — | Absolute path to a Chrome/Chromium binary. Empty = find one. |
| `browser.headless` | `ESCALATOR_BROWSER_HEADLESS` | `true` | false needs a display (or Xvfb), and is harder to detect. |
| `browser.via_proxy` | `ESCALATOR_BROWSER_VIA_PROXY` | `true` | Route renders through the proxy too. Costs bandwidth. |
| `browser.max_concurrent` | `ESCALATOR_BROWSER_MAX_CONCURRENT` | `4` | Chrome is the RAM ceiling on a small box. |
| `browser.timeout_ms` | `ESCALATOR_BROWSER_TIMEOUT_MS` | `30000` | Per-fetch deadline for the browser rung. |
| `proxy.enabled` | `ESCALATOR_PROXY_ENABLED` | `false` | The switch. Everything below is ignored while this is false. |
| `proxy.url` | `ESCALATOR_PROXY_URL` | — | http://user:pass@host:port, or socks5://... |
| `proxy.list` | `ESCALATOR_PROXY_LIST` | — | Several exits, used round-robin. Combined with url. |
| `http.timeout_ms` | `ESCALATOR_HTTP_TIMEOUT_MS` | `10000` | Per-fetch deadline for the two http rungs. |
| `ladder.min_content_chars` | `ESCALATOR_LADDER_MIN_CONTENT_CHARS` | `200` | Below this many extracted chars a page is 'thin' and the ladder climbs. |
| `ladder.tier_cache_ttl_hours` | `ESCALATOR_LADDER_TIER_CACHE_TTL_HOURS` | `72` | How long a learned rung survives before decaying one step cheaper. |
| `politeness.respect_robots` | `ESCALATOR_POLITENESS_RESPECT_ROBOTS` | `true` | Your box, your call. |
| `politeness.rate_limit_rps` | `ESCALATOR_POLITENESS_RATE_LIMIT_RPS` | `1.0` | Per-domain. 0 disables the gap entirely. |
| `politeness.user_agent` | `ESCALATOR_POLITENESS_USER_AGENT` | a Chrome UA | Used for robots.txt matching. |
| `server.api_keys` | `ESCALATOR_SERVER_API_KEYS` | — | Bearer keys for `escalator serve`. This list IS the truth: removing one revokes it. |
| `server.host` | `ESCALATOR_SERVER_HOST` | `127.0.0.1` | 127.0.0.1 keeps it off the local network. Containers/services behind a proxy want it here; only face the internet through a reverse proxy. |
| `server.port` | `ESCALATOR_SERVER_PORT` | `8000` | Port for `escalator serve`. |
| `storage.data_dir` | `ESCALATOR_STORAGE_DATA_DIR` | — | Database and managed browsers. Empty = the platform default. |
| `storage.request_log_limit` | `ESCALATOR_STORAGE_REQUEST_LOG_LIMIT` | `5000` | Rows kept in request_log; trimmed on insert. |

Data — the SQLite database and any downloaded browser — lives in the platform
data dir, overridable with `ESCALATOR_STORAGE_DATA_DIR`. Nothing is ever written
outside it.

### Where the browser comes from

`escalator browser list` shows the search, in order:

1. an explicit path — `--browser-path`, then `ESCALATOR_BROWSER_PATH`, then
   `browser.path`. If set and wrong, that's an error naming the path, never a
   silent fall-through.
2. browsers installed on this machine: real Google Chrome first, then Chromium,
   then Edge and Brave.
3. a browser `escalator browser install` downloaded earlier.

If none of those find anything, you get an error naming the two commands that fix
it. Resolution never downloads on its own — a server request or a cron job should
not install software as a side effect.

## Using the API

```bash
escalator serve   # binds ESCALATOR_SERVER_HOST:PORT (127.0.0.1:8000 by default)
```

```
POST /scrape   {url, markdown?, min_tier?, max_tier?, timeout_ms?}  -> FetchResult
GET  /healthz                                                       -> {status, version}
```

Authenticate every `/scrape` call with `Authorization: Bearer <key>`, where the
key is one of `server.api_keys`. That list *is* the truth: remove a key and it is
revoked on the next start. Mint one with `escalator key generate`; there is
deliberately no *HTTP* endpoint to do so — a running server should not be able to
grant itself credentials. `/healthz` needs no auth — it's what a load balancer
polls.

```bash
curl -s http://127.0.0.1:8000/scrape \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}'
```

A wall comes back as `200 OK` with `{"status": "challenged"}`, not an HTTP error
— deliberately, so an agent on the other end can react. Retrying in a loop won't
help: escalator does not solve CAPTCHAs, by design. A residential proxy is the
usual answer.

With the `[mcp]` extra the same ladder is exposed at `/mcp` as one tool,
`scrape_url(url, force_browser=False)`, for MCP-speaking agents.

## Command reference

```
escalator init [--yes]     configure this machine; --yes for scripts
escalator doctor [--json]  check everything, one fix per failure
escalator browser list     every browser found, and which one wins
escalator browser install  download Chrome for Testing into the data dir
escalator key generate     mint a bearer key into the config (--show: print only)
escalator scrape URL       one page to stdout, so it pipes (--html for raw HTML)
escalator serve            the HTTP API and the MCP face
escalator --version        tool, python, platform
```

## The ladder

```
policy      robots.txt (cached) + rate limit  → may short-circuit (skip/deny/wait)
http        curl_cffi, impersonate=chrome     → ~100ms; clears undefended sites
http_proxy  same, routed via residential IP   → beats datacenter-IP bans
browser     nodriver, headless Chrome         → JS/SPA + Cloudflare-class defenses
                    │
                    └─ walled on the last rung? → status="challenged". Surrender.
```

Two things make this more than a `for` loop:

**200 OK is not success.** A rung that returns HTTP 200 carrying a Cloudflare
interstitial has not succeeded. `core/detect.py` classifies every response
*after* extraction — `content`, `thin`, or `blocked` — and only `content` counts.
Without that the ladder would never escalate, and the cache would learn "http
works" for a domain that serves junk forever.

**The cache forgets.** A learned start-rung that only ratcheted upward would
drift every domain toward browser+proxy and quietly inflate your proxy bill.
Entries carry `learned_at`; past `ladder.tier_cache_ttl_hours` a domain retries
one rung cheaper.

See [DESIGN.md](DESIGN.md) for why it is shaped this way — and for what it
deliberately refuses to do.

## Troubleshooting

Start here — always:

```bash
escalator doctor
```

Every ❌ comes with the one line that fixes it. Common cases:

| symptom | what it usually is |
|---|---|
| `no Chrome-family browser found` | `escalator browser install` |
| `error while loading shared libraries` on Linux | you skipped [step 1](#1-system-libraries-the-browser-needs); doctor names the exact package |
| everything returns `challenged` | you need a residential proxy: set `ESCALATOR_PROXY_ENABLED=true` and a `PROXY_URL` |
| `/scrape` returns 401 | no key configured, or it was removed from `server.api_keys` |
| container crashes on heavy pages | raise `--shm-size` (compose: `shm_size: 512mb`) |
| bind-mounted `/data` is read-only in Docker | `chown -R 1000:1000 ./data`, or use a named volume |
| slow first browser fetch | Chrome cold start; escalator retries the launch once |

If it still won't work, paste the whole `escalator doctor` output into an issue —
that's what its last line asks for, and the fastest route to an answer.

## Development

See [CONTRIBUTING.md](CONTRIBUTING.md). In short: `uv sync`, `uv run pytest`.

## License

Released into the public domain — see [UNLICENSE](UNLICENSE). No warranty, no
attribution required, do what you like with it.
</content>
</invoke>