# Proxy Browser — MCP Server

An [MCP](https://modelcontextprotocol.io) server that lets Claude browse the web and make
**GET / POST / PUT / PATCH / DELETE / HEAD** requests while looking like a real **Chrome**
browser at the network layer — so you don't get anti-bot flagged on sites like AliExpress.

- Real Chrome TLS/HTTP2 fingerprint (via `curl_cffi`), not just a spoofed User-Agent
- **Headless-browser render tier** — falls back to a real, undetected Chromium (patchright)
  that executes JavaScript and passes JS anti-bot challenges (Cloudflare, DataDome, AliExpress)
- **Anti-bot challenge detection** — flags captcha/punish walls instead of returning the stub as content
- Persistent cookie session — multi-step browsing looks like one continuous browser
- Token-efficient output — HTML is cleaned into markdown by default (raw available)
- Streamed, byte-capped body reads — an oversized/binary response can't blow up memory
- Automatic retries with backoff + jitter (honors `Retry-After`) for idempotent methods
- Randomized think-time + a hard 1–2 request cap per call — a constant, bursty cadence is itself a bot tell
- Optional upstream proxy — plug in a residential/rotating proxy for IP rotation
- Full verb set with form or JSON bodies

---

## Why not just `requests` + a Chrome User-Agent?

Because a UA string fools nobody serious. Cloudflare, Akamai, and AliExpress fingerprint
your **TLS handshake (JA3/JA4)** and **HTTP/2 frame ordering** — a stock Python HTTP client
is instantly identifiable as a bot no matter what `User-Agent` header it sends.

This server uses [`curl_cffi`](https://github.com/lexiforest/curl_cffi), which drives
libcurl with a **real Chrome TLS + HTTP/2 fingerprint**. The connection itself looks like
Chrome — verified against `tls.peet.ws` (JA4 `t13d1516h2`, Chrome's Akamai HTTP/2
fingerprint, over `h2`).

---

## Requirements

- **Python 3.9+** (developed/tested on 3.11, Windows 11)
- **pip**
- **Claude Code** CLI (to consume the MCP server)
- Python packages (installed via `requirements.txt`):

  | Package          | Role                                                    |
  | ---------------- | ------------------------------------------------------- |
  | `mcp`            | Official MCP Python SDK (FastMCP server)                |
  | `curl_cffi`      | Chrome TLS/HTTP2 impersonation engine (bundled libcurl) |
  | `trafilatura`    | Main-content extraction → markdown (token-efficient)    |
  | `beautifulsoup4` | Full-page DOM cleanup                                   |
  | `markdownify`    | HTML → markdown conversion                               |

`curl_cffi` ships its own libcurl build, so there is **no external C dependency** to install.

---

## Installation

```powershell
# 1. Clone the repo
git clone https://github.com/nullsection/headless-proxy-mcp.git
cd headless-proxy-mcp

# 2. (recommended) create a virtual environment
python -m venv .venv
.\.venv\Scripts\Activate.ps1        # PowerShell
# source .venv/bin/activate         # macOS/Linux

# 3. Install dependencies
python -m pip install -r requirements.txt
```

---

## Setup — register with Claude Code

```powershell
# User scope = available in all your projects
claude mcp add -s user proxy-browser -- python C:/path/to/headless-proxy-mcp/proxy_browser_server.py

# Verify
claude mcp get proxy-browser        # should show: Status: Connected
```

> Use forward slashes in the path — a backslash path can be mangled by the shell.

To pass configuration (e.g. a proxy), add `-e` flags:

```powershell
claude mcp add -s user proxy-browser \
  -e PROXY_URL=http://user:pass@host:port \
  -e IMPERSONATE=chrome124 \
  -- python C:/path/to/headless-proxy-mcp/proxy_browser_server.py
```

**Restart Claude Code after registering** so it loads the server.

---

## Usage

Once registered and restarted, just ask Claude in natural language — e.g.
*"browse aliexpress for usb cables and summarize the top results."* Claude selects the
right tool automatically.

### Tools

| Tool            | Description                                                          |
| --------------- | ------------------------------------------------------------------- |
| `browse`        | HTTP **GET** a URL, with an optional headless-browser fallback (`render` policy). |
| `render`        | Load a URL in a **real headless browser** (executes JS; passes JS challenges).    |
| `http_post`     | HTTP **POST** with form `data` or `json` body.                      |
| `http_put`      | HTTP **PUT** with form `data` or `json` body.                       |
| `http_patch`    | HTTP **PATCH** with form `data` or `json` body.                     |
| `http_delete`   | HTTP **DELETE** (optional `data` or `json` body).                   |
| `http_head`     | HTTP **HEAD** — response headers only (liveness / type / redirect). |
| `session_reset` | Clear cookies / start a fresh browsing identity.                    |
| `session_info`  | Show impersonation target, proxy, config, and current cookies.      |

> **Retries & idempotency:** `browse`, `http_put`, `http_delete`, and `http_head` are
> idempotent, so transient `429`/`5xx`/network failures are retried automatically with
> exponential backoff (honoring `Retry-After`). `http_post` and `http_patch` are **not**
> retried, to avoid duplicate side effects.

### Parameters

- **`url`** — full URL including `http://` or `https://`.
- **`extract`** — how the body is returned:
  - `"auto"` (default) — main content as markdown (best for pages)
  - `"full"` — the whole page converted to markdown
  - `"text"` — plain text only
  - `"raw"` — exact HTML / response bytes
- **`headers`** — optional extra request headers (merged over sane Chrome defaults).
- **`proxy`** — optional per-request proxy URL (overrides `PROXY_URL` for that call).
- **`timeout`** — request timeout in seconds (default 30).
- **`data`** / **`json`** (POST/PUT only) — form fields **or** a JSON body (use one).

---

## Configuration

Set via environment variables (or `-e` flags at registration). See `.env.example`.

| Variable             | Default     | Description                                                        |
| -------------------- | ----------- | ------------------------------------------------------------------ |
| `IMPERSONATE`        | `chrome131` | Chrome build to mimic (e.g. `chrome124`, `chrome131`, `chrome136`). Keep it recent. |
| `PROXY_URL`          | *(empty)*   | Upstream proxy: `http://user:pass@host:port` or `socks5://host:port`. Empty = direct. |
| `MAX_CHARS`          | `60000`     | Max characters of the **extracted** body returned to Claude (protects context). |
| `MAX_RESPONSE_BYTES` | `10000000`  | Hard cap on **raw** bytes read off the wire; body is streamed and reading stops past this. |
| `PER_HOST_DELAY`     | `0`         | Minimum seconds between requests to the same host (politeness / anti-flag). |
| `PER_HOST_JITTER`    | `0.3`       | Randomize the per-host delay by ±this fraction (0.3 = ±30%) so cadence isn't constant. |
| `MAX_RETRIES`        | `2`         | Additional attempts after the first on `429`/`5xx`/network errors (bounded by `MAX_REQUESTS_PER_CALL`). |
| `RETRY_BACKOFF`      | `0.5`       | Base retry backoff in seconds, grown exponentially with jitter. |
| `MAX_RETRY_WAIT`     | `30`        | Cap on any single retry sleep, even if `Retry-After` asks for more. |
| `REQUEST_JITTER_MIN` / `REQUEST_JITTER_MAX` | `0.5` / `2.0` | Randomized think-time (s) before **every** request — fires even on a single hit, so nothing goes out with bot-like zero latency. `MAX=0` disables. |
| `MAX_REQUESTS_PER_CALL` | `2`      | Hard ceiling on total HTTP hits per tool call (initial + retries), clamped to **1–2** to stay under anti-bot thresholds. |
| `RENDER_MODE`        | `auto`      | Browser-fallback policy for `browse`: `auto` \| `never` \| `always`. |
| `RENDER_WAIT_UNTIL`  | `domcontentloaded` | Load state renders wait for; use `networkidle` for XHR-driven grids. |
| `RENDER_HEADLESS`    | `1`         | `0` = headful window (max stealth, needs a display). |
| `RENDER_CHANNEL`     | *(bundled)* | `chrome`/`msedge` to drive the real installed browser (new headless, real GPU). Recommended for hard walls. |
| `RENDER_USER_DATA_DIR` | *(none)*  | Persistent Chrome profile dir — banks anti-bot cookies + logins across runs (the PerimeterX lever). |
| `RENDER_EXTRA_ARGS`  | *(none)*    | Extra Chromium flags, comma-separated (e.g. `--use-angle=d3d11` for real-GPU WebGL on Windows). |
| `RENDER_LOCALE` / `RENDER_TIMEZONE` / `RENDER_VIEWPORT` / `RENDER_UA` | `en-US` / `America/New_York` / `1280x800` / *(auto)* | Realistic browser context. UA defaults to the engine's real UA with the `HeadlessChrome` tell stripped. |

See `.env.example` for the full render knob list (`RENDER_TIMEOUT`, `RENDER_WAIT_MS`, `RENDER_MIN_CONTENT_CHARS`).

---

## Troubleshooting

- **`Failed to connect`** — the path was likely mangled by backslashes. Re-add using
  **forward slashes** (see setup above), and confirm `python` is on your PATH.
- **Server connects but a site still blocks you** — you're probably being flagged by **IP**,
  not fingerprint. Set `PROXY_URL` to a residential/rotating proxy and raise `PER_HOST_DELAY`.
- **Page comes back blank / missing content** — the site renders via JavaScript. `curl_cffi`
  does not execute JS (see Limitations).
- **Response truncated** — raise `MAX_CHARS`, or request a tighter `extract` mode.

---

## Headless browser render tier

The HTTP engine mimics Chrome's *network identity* but can't run JavaScript, so JS-rendered
pages and JS challenge walls defeat it. The optional render tier fixes that by driving a
**real, undetected Chromium** ([patchright](https://github.com/Kaliiiiiiiiii-Vinyzu/patchright),
a patched Playwright drop-in) that executes JS, passes challenges, and hides the automation
tells (`navigator.webdriver`, the `HeadlessChrome` UA).

```powershell
pip install -r requirements-render.txt
patchright install chromium          # one-time browser download (or: playwright install chromium)
```

- **`browse(url, render="auto")`** (default) — tries fast HTTP first, then transparently
  escalates to the browser when the page looks challenge-walled or JS-thin **and** an engine
  is installed. `render="never"` = HTTP only; `render="always"` = browser only.
- **`render(url, wait_selector=".product-card", wait_ms=4000)`** — explicit browser load.
  Use `wait_selector`/`wait_ms` (or `RENDER_WAIT_UNTIL=networkidle`) for grids that populate
  via XHR after load.

> Verified live: this tier passes AliExpress's `_tmd_/punish` captcha wall that blocks the HTTP
> client, and renders the JS product grid. `navigator.webdriver` is hidden and the UA is cleaned
> to a real Chrome string.

### Getting past hard walls (PerimeterX / DataDome / Facebook)

For aggressive bot managers, the render tier has a **persistent-profile + organic-navigation**
mode. Verified live getting past **Carsales' PerimeterX** wall (which `403`s a cold session):

- `RENDER_CHANNEL=chrome` + `RENDER_USER_DATA_DIR=<dedicated dir>` + `RENDER_EXTRA_ARGS=--use-angle=d3d11`
- `render(url, warm=[...homepage, category...])` — step down to the deep page so the anti-bot
  cookie matures instead of cold-linking.
- `render(url, press=["Escape"])` — dismiss **Facebook Marketplace's** login overlay.

**See [docs/anti-bot-bypass.md](docs/anti-bot-bypass.md) for the full playbook** — the wall tiers,
the detection-signal table, the exact PerimeterX recipe, pacing/cooldown notes, and when to reach
for a managed unblocker instead.

## Testing

Unit tests are **hermetic** — they stub `curl_cffi`/`mcp` and mock the browser, so no native
libs, browser, or network are needed. They cover the request/retry loop, the streamed byte cap,
`Retry-After` handling, challenge detection, extraction dispatch, the tiered render fallback,
and the render engine (including its asyncio-safety) — **~92% line coverage across both modules**.

```powershell
python -m pip install -r requirements-dev.txt
python -m pytest tests/                       # fast, offline
python -m coverage run -m pytest tests/ ; python -m coverage report proxy_browser_server.py
```

### Live smoke test (AliExpress)

`tests/smoke_aliexpress.py` drives the **real** server against AliExpress — exercising the
full Chrome-impersonation path, retries, byte cap, and HTML→markdown extraction end to end.
Run it from the same env the MCP runs in (needs the runtime deps installed):

```powershell
python tests/smoke_aliexpress.py                       # "moza racing steering wheel", cheapest-first
python tests/smoke_aliexpress.py "moza r9 wheelbase"   # custom query
python tests/smoke_aliexpress.py --raw "moza ks wheel" # raw HTML instead of markdown
```

The same check runs under pytest when opted in:

```powershell
$env:RUN_SMOKE=1 ; python -m pytest tests/test_smoke.py::test_live_aliexpress_moza
```

> AliExpress is heavily JS-rendered and anti-bot defended. The HTTP path gets a real `200` but
> thin/challenge HTML; the [render tier](#headless-browser-render-tier) passes the wall and
> returns the full product grid.

### Live render test (opt-in)

With the render deps installed, exercise the real browser end to end:

```powershell
$env:RUN_RENDER=1 ; python -m pytest tests/test_browser_engine.py::test_live_render_executes_js_and_returns_html
```

## Limitations

- **JavaScript** is handled by the [render tier](#headless-browser-render-tier), **not** the
  HTTP engine. Plain `curl_cffi` can't run JS; install the render deps (patchright + Chromium)
  and use `render="auto"`/`render="always"` for JS-rendered or challenge-walled sites. Without
  the render deps installed, JS-heavy pages come back thin (and are flagged when it's a known
  anti-bot wall).
- **Render tier is heavier.** A browser launch costs ~1–3 s and real memory; it only fires on
  demand (challenge/thin page in `auto`, or explicit `render`), not on every request.
- **IP reputation still matters.** Even the browser can be IP-flagged after abuse; for the
  most aggressive sites pair with a residential `PROXY_URL` and/or `RENDER_HEADLESS=0`.
- **Respect Terms of Service and rate limits.** This tool is for legitimate automation and
  research; the built-in per-host throttle, request jitter, and 1–2 request cap keep it polite —
  don't hammer sites.

---

## Project layout

```
headless-proxy-mcp/
├── proxy_browser_server.py   # the MCP server — HTTP engine, tools, tiered fallback
├── browser_engine.py         # headless-browser render engine (patchright/playwright)
├── requirements.txt          # runtime dependencies (HTTP engine)
├── requirements-render.txt   # optional: headless browser render tier
├── requirements-dev.txt      # test dependencies (pytest, coverage)
├── .env.example              # configuration reference
├── .gitignore
├── README.md
├── docs/
│   └── anti-bot-bypass.md    # playbook: wall tiers, PerimeterX recipe, FB trick
└── tests/
    ├── conftest.py            # stubs curl_cffi/mcp so tests run offline
    ├── test_proxy_browser.py  # hermetic unit tests — HTTP engine + tiered fallback
    ├── test_browser_engine.py # hermetic unit tests — render engine (mocked browser)
    ├── smoke_aliexpress.py    # live smoke test runner (real network)
    └── test_smoke.py          # smoke-test unit coverage + opt-in live check
```

## License

MIT (or your preference).