SkyNet Scraper

Free web search and page scraping for any AI coding agent. No API keys, no accounts, no configuration.

SkyNet runs a real browser (Botasaurus' anti-detect driver over CDP), which is why it works where plain HTTP clients get a captcha page. It ships as three things from one package: an MCP server, a CLI, and a Python module.

agent ──stdio JSON-RPC──> skynet ──CDP──> Chrome ──> web

MIT licensed · Python 3.9+ · macOS and Linux · requires Google Chrome

Install

One command, any machine with Python 3.9+ and Chrome:

uv tool install git+https://github.com/zyetaone/skynet-scraper

No uv? Use pipx install git+https://github.com/zyetaone/skynet-scraper, or pip install into a virtualenv.

Then register it in your agent apps — also one command:

skynet install

It writes the right config for every app it finds (OpenCode, Claude Desktop, Cursor, VS Code, Codex), backs up nothing it doesn't have to, and skips apps that aren't installed. Restart the app and you're done.

skynet doctor

skynet    1.0.0
python    3.14.10
driver    botasaurus-driver 4.0.101
chrome    found
search test → engine duckduckgo → 2 result(s)

Related MCP server: browser-search-mcp

Quick start

As an agent — ask for anything that needs the web; the agent calls:

From a shell:

skynet search "rust 1.99 release"
skynet search "openspace api" --limit 5 --engine bing
skynet scrape https://doc.rust-lang.org/book/
skynet scrape https://example.com --format links
skynet scrape https://example.com --selector "article" --max-chars 4000
skynet scrape https://challenged.example/ --fallback https://challenged.example/api.json

Every command prints JSON and exits non-zero when nothing came back, so scripts can branch on it.

From Python:

from skynet_scraper import server

results = server.search("rust release notes", limit=5)
doc = server.page("https://doc.rust-lang.org/book/", format="text", max_chars=4000)

Configuration

Everything is optional; there are no keys to obtain.

Tool arguments override the environment per call.

How it works

• One shared browser. The first call launches Chrome with a small profile; later calls reuse it. That is what makes a scrape take a couple of seconds instead of ten.

• Search opens the engine's results page in that browser and parses the SERP with BeautifulSoup. engine: auto walks DuckDuckGo → DDG-lite → Bing → Mojeek and returns which one answered, plus an attempts list when all of them fail.

• Page opens the URL, waits for it, and extracts either readable text (scripts/styles stripped), raw HTML, or links. --selector narrows to one CSS block.

• Protocol safety. stdout carries only JSON-RPC; library and browser output is redirected to stderr. The read loop uses readline(), not iteration, so clients that keep stdin open don't stall.

• No daemon. Nothing runs in the background. Your agent spawns the server on demand.

When a site serves a challenge

page recognises bot-challenge pages (Client Challenge, Just a moment..., Attention Required, checking your browser, verify you are human, cf-challenge, ddos-guard) and never passes them off as content. It then, in order:

1. Uses your fallback URL — JSON is returned parsed (json, or json_text when larger than max_chars), HTML is fetched normally.

2. Serves the closest Wayback snapshot, flagged "archived": true with a staleness note.

3. Reports "challenge": "<marker>" with a hint.

Challenges are mostly IP reputation; the durable fix is BOTASAURUS_MCP_PROXY.

Limitations, stated plainly

• Search engines' terms of service forbid scraping. This is a tool for your own engineering work, not a public search API. Keep volume sane or you will be challenged.

• Cloudflare can still refuse you. Anti-detect is a moving target, not a guarantee; fallback and proxies are the reliable answers.

• Engine markup changes. When nothing parses, search reports per-engine attempts; fix the selectors in _selectors() in src/skynet_scraper/server.py.

• It uses the Chrome already installed. A machine policy that blocks Chrome kills this too — skynet doctor tells you immediately.

• Not a crawler. No queue, no scheduling, no robots.txt handling, no deduplication across runs. Add what you need on top; the API is small on purpose.

• No JavaScript interaction. It reads pages; it does not click, fill forms or log in. Use Playwright or the app's own browser tools for that.

Contributing

Contributions are welcome — see CONTRIBUTING.md. Short version:

git clone https://github.com/zyetaone/skynet-scraper
cd skynet-scraper
python3 -m venv .venv && . .venv/bin/activate
pip install -e .
python -m unittest discover -s tests -v

The parsing layer is pure and fully tested offline, so most changes need no browser.

Where to build next

Good, self-contained places to extend:

• New search engines — add a URL template to ENGINE_URLS and a selector row to _selectors(); add a fixture test.

• Richer extraction — page already supports CSS selectors; a format: "markdown" would make it agent-friendlier.

• Challenge strategies — a proxy pool, cookie reuse, or per-domain fallbacks.

• MCP resources/prompts — the server implements tools only; resources are easy to add.

• Pagination — search returns one page; offset/page arguments are a natural next step.

License

MIT © Binae. See LICENSE.