Proxy Browser — MCP Server

An MCP server that lets Claude browse the web and make GET / POST / PUT / PATCH / DELETE / HEAD requests while looking like a real Chrome browser at the network layer — so you don't get anti-bot flagged on sites like AliExpress.

• Real Chrome TLS/HTTP2 fingerprint (via curl_cffi), not just a spoofed User-Agent

• Headless-browser render tier — falls back to a real, undetected Chromium (patchright) that executes JavaScript and passes JS anti-bot challenges (Cloudflare, DataDome, AliExpress)

• Anti-bot challenge detection — flags captcha/punish walls instead of returning the stub as content

• Persistent cookie session — multi-step browsing looks like one continuous browser

• Token-efficient output — HTML is cleaned into markdown by default (raw available)

• Streamed, byte-capped body reads — an oversized/binary response can't blow up memory

• Automatic retries with backoff + jitter (honors Retry-After) for idempotent methods

• Randomized think-time + a hard 1–2 request cap per call — a constant, bursty cadence is itself a bot tell

• Optional upstream proxy — plug in a residential/rotating proxy for IP rotation

• Full verb set with form or JSON bodies

Why not just requests + a Chrome User-Agent?

Because a UA string fools nobody serious. Cloudflare, Akamai, and AliExpress fingerprint your TLS handshake (JA3/JA4) and HTTP/2 frame ordering — a stock Python HTTP client is instantly identifiable as a bot no matter what User-Agent header it sends.

This server uses curl_cffi, which drives libcurl with a real Chrome TLS + HTTP/2 fingerprint. The connection itself looks like Chrome — verified against tls.peet.ws (JA4 t13d1516h2, Chrome's Akamai HTTP/2 fingerprint, over h2).

Related MCP server: gotham-browser

Requirements

• Python 3.9+ (developed/tested on 3.11, Windows 11)

• pip

• Claude Code CLI (to consume the MCP server)

• Python packages (installed via requirements.txt):

  Package	Role
  mcp	Official MCP Python SDK (FastMCP server)
  curl_cffi	Chrome TLS/HTTP2 impersonation engine (bundled libcurl)
  trafilatura	Main-content extraction → markdown (token-efficient)
  beautifulsoup4	Full-page DOM cleanup
  markdownify	HTML → markdown conversion

curl_cffi ships its own libcurl build, so there is no external C dependency to install.

Installation

# 1. Clone the repo
git clone https://github.com/nullsection/headless-proxy-mcp.git
cd headless-proxy-mcp

# 2. (recommended) create a virtual environment
python -m venv .venv
.\.venv\Scripts\Activate.ps1        # PowerShell
# source .venv/bin/activate         # macOS/Linux

# 3. Install dependencies
python -m pip install -r requirements.txt

Setup — register with Claude Code

# User scope = available in all your projects
claude mcp add -s user proxy-browser -- python C:/path/to/headless-proxy-mcp/proxy_browser_server.py

# Verify
claude mcp get proxy-browser        # should show: Status: Connected

Use forward slashes in the path — a backslash path can be mangled by the shell.

To pass configuration (e.g. a proxy), add -e flags:

claude mcp add -s user proxy-browser \
  -e PROXY_URL=http://user:pass@host:port \
  -e IMPERSONATE=chrome124 \
  -- python C:/path/to/headless-proxy-mcp/proxy_browser_server.py

Restart Claude Code after registering so it loads the server.

Usage

Once registered and restarted, just ask Claude in natural language — e.g. "browse aliexpress for usb cables and summarize the top results." Claude selects the right tool automatically.

Tools

Retries & idempotency: browse, http_put, http_delete, and http_head are idempotent, so transient 429/5xx/network failures are retried automatically with exponential backoff (honoring Retry-After). http_post and http_patch are not retried, to avoid duplicate side effects.

Parameters

• url — full URL including http:// or https://.

• extract — how the body is returned:

  • "auto" (default) — main content as markdown (best for pages)

  • "full" — the whole page converted to markdown

  • "text" — plain text only

  • "raw" — exact HTML / response bytes

• headers — optional extra request headers (merged over sane Chrome defaults).

• proxy — optional per-request proxy URL (overrides PROXY_URL for that call).

• timeout — request timeout in seconds (default 30).

• data / json (POST/PUT only) — form fields or a JSON body (use one).

Configuration

Set via environment variables (or -e flags at registration). See .env.example.

See .env.example for the full render knob list (RENDER_TIMEOUT, RENDER_WAIT_MS, RENDER_MIN_CONTENT_CHARS).

Troubleshooting

• Failed to connect — the path was likely mangled by backslashes. Re-add using forward slashes (see setup above), and confirm python is on your PATH.

• Server connects but a site still blocks you — you're probably being flagged by IP, not fingerprint. Set PROXY_URL to a residential/rotating proxy and raise PER_HOST_DELAY.

• Page comes back blank / missing content — the site renders via JavaScript. curl_cffi does not execute JS (see Limitations).

• Response truncated — raise MAX_CHARS, or request a tighter extract mode.

Headless browser render tier

The HTTP engine mimics Chrome's network identity but can't run JavaScript, so JS-rendered pages and JS challenge walls defeat it. The optional render tier fixes that by driving a real, undetected Chromium (patchright, a patched Playwright drop-in) that executes JS, passes challenges, and hides the automation tells (navigator.webdriver, the HeadlessChrome UA).

pip install -r requirements-render.txt
patchright install chromium          # one-time browser download (or: playwright install chromium)

• browse(url, render="auto") (default) — tries fast HTTP first, then transparently escalates to the browser when the page looks challenge-walled or JS-thin and an engine is installed. render="never" = HTTP only; render="always" = browser only.

• render(url, wait_selector=".product-card", wait_ms=4000) — explicit browser load. Use wait_selector/wait_ms (or RENDER_WAIT_UNTIL=networkidle) for grids that populate via XHR after load.

Verified live: this tier passes AliExpress's _tmd_/punish captcha wall that blocks the HTTP client, and renders the JS product grid. navigator.webdriver is hidden and the UA is cleaned to a real Chrome string.

Getting past hard walls (PerimeterX / DataDome / Facebook)

For aggressive bot managers, the render tier has a persistent-profile + organic-navigation mode. Verified live getting past Carsales' PerimeterX wall (which 403s a cold session):

• RENDER_CHANNEL=chrome + RENDER_USER_DATA_DIR=<dedicated dir> + RENDER_EXTRA_ARGS=--use-angle=d3d11

• render(url, warm=[...homepage, category...]) — step down to the deep page so the anti-bot cookie matures instead of cold-linking.

• render(url, press=["Escape"]) — dismiss Facebook Marketplace's login overlay.

See docs/anti-bot-bypass.md for the full playbook — the wall tiers, the detection-signal table, the exact PerimeterX recipe, pacing/cooldown notes, and when to reach for a managed unblocker instead.

Testing

Unit tests are hermetic — they stub curl_cffi/mcp and mock the browser, so no native libs, browser, or network are needed. They cover the request/retry loop, the streamed byte cap, Retry-After handling, challenge detection, extraction dispatch, the tiered render fallback, and the render engine (including its asyncio-safety) — ~92% line coverage across both modules.

python -m pip install -r requirements-dev.txt
python -m pytest tests/                       # fast, offline
python -m coverage run -m pytest tests/ ; python -m coverage report proxy_browser_server.py

Live smoke test (AliExpress)

tests/smoke_aliexpress.py drives the real server against AliExpress — exercising the full Chrome-impersonation path, retries, byte cap, and HTML→markdown extraction end to end. Run it from the same env the MCP runs in (needs the runtime deps installed):

python tests/smoke_aliexpress.py                       # "moza racing steering wheel", cheapest-first
python tests/smoke_aliexpress.py "moza r9 wheelbase"   # custom query
python tests/smoke_aliexpress.py --raw "moza ks wheel" # raw HTML instead of markdown

The same check runs under pytest when opted in:

$env:RUN_SMOKE=1 ; python -m pytest tests/test_smoke.py::test_live_aliexpress_moza

AliExpress is heavily JS-rendered and anti-bot defended. The HTTP path gets a real 200 but thin/challenge HTML; the render tier passes the wall and returns the full product grid.

Live render test (opt-in)

With the render deps installed, exercise the real browser end to end:

$env:RUN_RENDER=1 ; python -m pytest tests/test_browser_engine.py::test_live_render_executes_js_and_returns_html

Limitations

• JavaScript is handled by the render tier, not the HTTP engine. Plain curl_cffi can't run JS; install the render deps (patchright + Chromium) and use render="auto"/render="always" for JS-rendered or challenge-walled sites. Without the render deps installed, JS-heavy pages come back thin (and are flagged when it's a known anti-bot wall).

• Render tier is heavier. A browser launch costs ~1–3 s and real memory; it only fires on demand (challenge/thin page in auto, or explicit render), not on every request.

• IP reputation still matters. Even the browser can be IP-flagged after abuse; for the most aggressive sites pair with a residential PROXY_URL and/or RENDER_HEADLESS=0.

• Respect Terms of Service and rate limits. This tool is for legitimate automation and research; the built-in per-host throttle, request jitter, and 1–2 request cap keep it polite — don't hammer sites.

Project layout

headless-proxy-mcp/
├── proxy_browser_server.py   # the MCP server — HTTP engine, tools, tiered fallback
├── browser_engine.py         # headless-browser render engine (patchright/playwright)
├── requirements.txt          # runtime dependencies (HTTP engine)
├── requirements-render.txt   # optional: headless browser render tier
├── requirements-dev.txt      # test dependencies (pytest, coverage)
├── .env.example              # configuration reference
├── .gitignore
├── README.md
├── docs/
│   └── anti-bot-bypass.md    # playbook: wall tiers, PerimeterX recipe, FB trick
└── tests/
    ├── conftest.py            # stubs curl_cffi/mcp so tests run offline
    ├── test_proxy_browser.py  # hermetic unit tests — HTTP engine + tiered fallback
    ├── test_browser_engine.py # hermetic unit tests — render engine (mocked browser)
    ├── smoke_aliexpress.py    # live smoke test runner (real network)
    └── test_smoke.py          # smoke-test unit coverage + opt-in live check

License

MIT (or your preference).