FrankenSurf

Every web tool breaks somewhere, and they break on different sites. Frankensurf sits in front of all of them and gives your agent one call. It tries the cheapest tool first, checks the page is real, moves to a stronger tool only when a site pushes back, and returns clean page data with a receipt of how it got there.

Quick start

Python 3.12+, on Linux, macOS, or Windows through WSL. No account or key needed.

pip install "frankensurf[mcp]"
frankensurf install        # every free tool: browsers, stealth browsers, local search, Crawl4AI
frankensurf read "https://www.walmart.com/search?q=air+fryer" --explain

The read climbs until it gets the real page:

https://www.walmart.com/search?q=air+fryer
  ✗ http                   CAPTCHA                0.4s
  ✗ local                  CAPTCHA                1.5s
  ✓ camoufox               got the page           15.6s

air fryer - Walmart.com
24,192 characters via camoufox, complete (107 results), free

frankensurf install works like playwright install: name just the tools you want (frankensurf install search chromium), add --with-deps for the browsers' system libraries on Linux, --list to see what's installed where, and frankensurf uninstall all to remove it. No Docker needed. Each release is installed from PyPI on fresh Ubuntu and macOS machines and checked.

Related MCP server: Deep Research MCP Server

Use it

From an agent, as an MCP server (setup for each client):

claude mcp add frankensurf -- frankensurf-mcp

From Python:

import frankensurf as fs

page = fs.read("https://example.com/pricing")
if page.ok:
    print(page.title, page.text[:500])
else:
    print("couldn't read it:", page.failure)    # {"code": "CAPTCHA", "message": ...}

pages = fs.batch(urls, links=True)               # paced per site, in order
hits = fs.search("playwright python tutorial", recency="month")
page = await fs.aread(url)                       # aread, abatch, asearch inside async code

For a long-running service, keep one Runtime open: async with Runtime() as web: await web.read(url).

Map and crawl: fs.map(url) lists a site's URLs; fs.crawl(url, limit=50) reads a section of it as markdown.

Agent mode (pip install "frankensurf[agent]", any LiteLLM model): fs.ask("question") searches, reads and answers with sources, and says what it couldn't read; fs.extract(urls, schema=...) and fs.summarize(url). Agent mode.

In Docker, one install for your whole network: docker run -p 3002:3002 -e FRANKENSURF_API_KEY=... ghcr.io/yail259/frankensurf (Docker).

From any tool that takes a Firecrawl URL (GPT Researcher, deep-research and others): frankensurf serve, then point it at http://127.0.0.1:3002 (setup).

From the command line:

frankensurf read https://example.com --markdown
frankensurf search playwright python tutorial --recency week
frankensurf watch https://news.ycombinator.com/ --link-pattern 'item\?id=\d+'

In Claude Code as a plugin (the MCP server plus the skill that teaches Claude to use it):

/plugin marketplace add yail259/frankensurf
/plugin install frankensurf@frankensurf

From LangChain, the OpenAI Agents SDK or CrewAI (pip install "frankensurf[langchain]", and so on):

from frankensurf.agent_tools import langchain_tools   # or openai_agents_tools, crewai_tools
tools = langchain_tools()                              # frankensurf_read, frankensurf_search

You get raw page data, not answers: text or markdown, JSON-LD, the page's own JSON, images, and the receipt. Your agent decides what the page means. If it isn't what it wanted, it asks again with retry_of=<trace_id> and Frankensurf skips every tool it already tried.

Benchmark

Cost per 1,000 correct pages: Frankensurf's free tools $0 in fees, about $0.01 of a $5/month server (87–88% right); with paid fallbacks $0.22 (96%); Firecrawl on its own $0.71 on its Scale plan, $0.98 on Standard (85%). How that's worked out.

scripts/toolbench.py runs each tool on its own, then Frankensurf, on the same pages with the same content checks. The newest set: 150 sites in 15 categories, picked before any was read, read once (11 October 2026, raw rows in benchmarks/2026-10-11/).

• Frankensurf read 17 sites Firecrawl missed; Firecrawl read none it missed.

• With no paid tools at all it is level with the best paid tool on its own (88.0% against 86.0% and 84.7%; at 150 sites that gap is within the noise).

• Its slowest reads are slower: the slowest tenth takes 25.7 s against Firecrawl's 10.6 s, when a read climbs several tools.

• Not a one-off: 95.3% vs 85.3% on the set the day before; 89.5% vs 80.3% and 88.5% vs 74.1% on earlier fresh sets.

Medians are over pages read; cost is per 1,000 pages read right. ZenRows and Scrapfly are left out (their plans ran out of credits). From a fresh cloud VM, whose data-centre address sites wall more often: free tools 82.0%, lightning mode 84.0% at a 1.2 s median. See Benchmarks.

Search

search uses free sources first and checks every result: on topic, the right kind of page, inside the date window. Exa, Brave, Tavily, Serper and other paid sources join only when you allow paid tools. Judged blind (results pooled and shuffled, which tool found them hidden), useful results in the top five:

Free, it beats Firecrawl's search: better on 35 queries, worse on 5. With paid allowed it is level with Exa. The local search engine (frankensurf install search, SearXNG without Docker) makes the free tier much stronger. v0.41 improved free search again: on a fresh blind set of 120 queries it beat v0.40 on 41 and lost on 12 (3.42 against 2.71 useful results of 5, benchmarks/2026-10-14-search-v041/). Details in benchmarks/2026-10-13-search/.

What else is in the box

• Every read carries receipt.quality: good, partial or poor, with flags such as paywall, menus or placeholder, so an agent knows when a page didn't really come through.

• --schema product.json (MCP schema) fills a JSON Schema from the page's JSON-LD, microdata, meta tags and own JSON into extracted, and climbs to a stronger tool while a required field is missing (Structured fields).

• --main (MCP main_content=True) adds main_text: the article without menus, footers and banners (best with pip install "frankensurf[main]").

• A read with no page after 3 seconds starts two more free tools beside it and keeps the first complete page (--lightning races from the start).

• A URL that answers "not found" is looked up with the site's own search; receipt.rescued says so.

• Still walled? Allow a real Chrome on your machine (--real-browser, with its own profile, never yours) or a second connection (FRANKENSURF_EGRESS).

• --site, --exclude-domain, --recency and --region on every source, and a --vertical (news, reference, discussions, qa, code, papers, books) backed by official free APIs.

• Monitoring-style queries ("Acme" OR "Acme Corp" (acquisition OR layoffs)) are judged by the name they're about, and a weak answer is retried once with just the name.

• --mode merge asks every source at once and fuses the results.

• frankensurf search-engine start|stop|status manages the local SearXNG; search restarts it after a reboot.

• A bundled skill teaches agents the tools: the MCP server sends it as its instructions, and frankensurf skill install adds it to ~/.claude/skills.

• Profiles: log in once and any tool can use the session; the model never sees cookies or keys.

• Human handoff: for a CAPTCHA or 2FA only a person can clear, it opens the page for you, then carries on.

• Site modules: save what your agent learns about a site as data. frankensurf module discover URL --query "desk lamp" --save finds the site's search, reads the results and saves a module; after that, searches on that site come back as structured items.

• Shared site knowledge: frankensurf share sync starts from which tool reads which site, as the community learned it; share export writes yours to review and contribute (opt in both ways).

• frankensurf watch-sites URL… --since DATE lists new pages from the feeds and sitemaps sites publish.

Everything works with no keys. Paid tools are opt-in and only come in once a site has walled a free tool, cheapest likely success first. Set their keys (for example FIRECRAWL_API_KEY) and pass --paid; --max-cost USD caps a read. From Python: allow_paid_fallbacks and max_cost_usd. See Paid tools and budgets.

How it works

Every way to fetch a page is a rung: plain HTTP and markdown negotiation, browsers, stealth browsers and signed requests, hosted browsers, paid APIs, and finally a person clearing the wall in a visible browser. A read starts at the cheapest rung and climbs only when a page pushes back. Every tool is a plugin, so new ones slot in as another rung.

See How escalation works and the full list of integrations.

Known walls

It's an alpha, and some sites still win:

• Need the paid tools: in a free-only spot check, Etsy, Home Depot, Yelp, Crunchbase and G2 blocked every free tool.

• Beat everything so far: Shopee, Temu and Idealista.

• Social sites without a login: LinkedIn, X, YouTube, Threads, Bluesky and Reddit read fine. Instagram, TikTok, Facebook and Pinterest show only the public shell; the posts need a login.

• Logged-in pages (your feeds, your orders) need a profile.

• A slow tail: most reads take a few seconds, but one in ten climbs several tools and takes 25 s or more.

Found a site it can't read? Report it; every report becomes a test case.

Develop

python3 -m venv .venv
.venv/bin/pip install -e '.[test,mcp]'
.venv/bin/frankensurf install
PYTHONPATH=.:src .venv/bin/pytest -q

The product design is in SPEC.rst and contributor rules in AGENTS.md. New tools are plugins, never branches in Core: see Writing a plugin and the example in examples/frankensurf-plugin-example. The website and docs live in site/.

Licence

Apache-2.0. See LICENSE.