spider-data

Web search, fetch, and crawl for AI agents — a drop-in replacement for built-in WebSearch/WebFetch, backed by the spider.cloud API.

Zero dependencies. One CLI, one MCP server, one skill file. Works with Claude Code, Codex, or anything that can run a subprocess or speak MCP.

spider-data search "postgres lock contention"
spider-data fetch  "https://example.com/article"
spider-data crawl  "https://docs.example.com" --limit 30

Why spider.cloud, wired this way

Cost-effective. Every fetch starts on a free direct request and only reaches the paid browser fleet on a confirmed block — a 403, a challenge page, an empty render target. There is no "this looks off, try the expensive one" branch, because that branch is how a web tool quietly runs up a bill. Results are cached (24 h search, 7 d fetch), revalidation is free, and follow-up reads — --section 3, --grep "timeout" — never cost anything. Each URL has a hard 2-paid-rung ceiling.

Reliable. Every outcome is a stable failure_code and a stable exit code, so an agent branches on exit 4 instead of reading a sentence and guessing. Blocked pages are detected structurally, not by vibes, so a short, Cloudflare-fronted, perfectly fine page doesn't get reported as a wall. Rate limits and 5xx retry with backoff and honour Retry-After. 184 tests, run on every publish.

Scalable. The cache is shared across runs and processes, so a fleet of agents researching the same topic pays once. A 14-day host memory learns which hosts always wall you and stops wasting the free rung on them — it can never add a paid call, only skip a doomed free one. Long pages return an outline plus a budgeted excerpt, so a big crawl doesn't blow up anyone's context window.

Related MCP server: superhighway-mcp

Install

Requires Node ≥20 or Bun. Nothing else.

git clone https://github.com/MirandaKim1434/spider-data
cd spider-data
npm link                      # or: ln -s "$PWD/bin/spider-data.js" ~/.local/bin/spider-data

SPIDER_API_KEY_INPUT=<your-key> spider-data auth login   # stores in the OS keychain
spider-data doctor

As a Claude Code skill

ln -s "$PWD/skill" ~/.claude/skills/spider-data

Then disable the built-ins in ~/.claude/settings.json, so there is one way to reach the web rather than two:

{
  "permissions": {
    "deny": ["WebSearch", "WebFetch"],
    "allow": ["Bash(spider-data:*)"]
  }
}

As an MCP server

{ "mcpServers": { "spider-data": { "command": "node", "args": ["/path/to/spider-data/src/mcp.js"] } } }

Exposes web_search, web_fetch, and web_crawl. See adapters/ for host-specific notes.

Commands

spider-data search <query>              spider-data cache {stats|clear|purge <url>}
spider-data fetch  <url>                spider-data stats [--since 7d]
spider-data crawl  <url>                spider-data doctor
                                        spider-data auth  {login|status}

Everywhere: --json for one machine-readable object, --refresh to ignore a cached copy, --debug to attach a redacted response snippet to errors.

Full list: spider-data --help.

Exit codes

Watching the spend

spider-data stats --since 7d

paid: true in the audit log means a request actually left the machine. Cache hits and free revalidations are never counted as paid, because the only function that can mark an attempt paid lives inside the function that opens the socket.

Security

The package holds an API key, so: it is never written into process.env, never read from a .env found by walking up the tree, and never allowed into an error message — any upstream can echo your request headers back at you. One redactor guards every write to stdout, stderr, the cache, and the log, and test/leak.test.js proves it against a server built to reflect headers back.

DNS is resolved and every address checked against the full private and reserved ranges before any request, so the paid API can't be used as an SSRF relay. robots.txt is honoured by default.

Tests

npm test          # or: bun test/run.js
npm run leakcheck # the key-leak suite on its own

184 tests, no network and no API key required. npm test also runs on publish.

More

docs/DESIGN.md covers how blocked pages are told apart from short ones, and what the package assumes rather than verifies.

Licence

MIT © Miranda Olson. See LICENSE.