# spider-data

Web search, fetch, and crawl for AI agents, backed by the
[spider.cloud](https://spider.cloud) API. Drop-in replacement for built-in
`WebSearch` and `WebFetch` tools.

Zero dependencies. One CLI, one MCP server, one skill file.

```bash
spider-data search "postgres lock contention"
spider-data fetch  "https://example.com/article"
spider-data crawl  "https://docs.example.com" --limit 30
```

## Why

**Cost-effective.** Fetches start on a free direct request. They escalate to the
paid browser fleet only on a confirmed block (403, challenge page, empty render
target), never on a guess. Each URL has a 2 paid-request ceiling. Results are
cached for 24 hours (search) and 7 days (fetch and crawl), revalidation is free,
and re-reads via `--section` and `--grep` cost nothing.

**Reliable.** Every outcome has a stable `failure_code` and exit code, so an
agent branches on `exit 4` instead of parsing prose. Block detection is
structural, not heuristic, so a short page behind a CDN is not misreported as
walled. Rate limits and 5xx responses retry with backoff and honour
`Retry-After`. 184 tests run on every publish.

**Scalable.** The cache is shared across runs and processes, so parallel agents
on the same topic pay once. A 14 day host memory skips the free request on hosts
that consistently return 403. It can only skip a free request, never add a paid
one. Long pages return a section outline plus a budgeted excerpt, keeping context
windows bounded on large crawls.

## Requirements

Node 20+ or Bun. No other dependencies.

## Install

```bash
git clone https://github.com/MirandaKim1434/spider-data
cd spider-data
npm link

SPIDER_API_KEY_INPUT=<your-key> spider-data auth login
spider-data doctor
```

`auth login` stores the key in the OS keychain.

On a machine with Bun but no Node, the `bin/` shebang will not resolve. Install a
wrapper instead of linking:

```bash
printf '#!/bin/sh\nexec bun "%s/bin/spider-data.js" "$@"\n' "$PWD" > ~/.local/bin/spider-data
chmod +x ~/.local/bin/spider-data
```

## Claude Code

```bash
ln -s "$PWD/skill" ~/.claude/skills/spider-data
```

Disable the built-ins in `~/.claude/settings.json` so there is one path to the
web rather than two:

```json
{
  "permissions": {
    "deny": ["WebSearch", "WebFetch"],
    "allow": ["Bash(spider-data:*)"]
  }
}
```

## MCP

```json
{ "mcpServers": { "spider-data": { "command": "node", "args": ["/path/to/spider-data/src/mcp.js"] } } }
```

Exposes `web_search`, `web_fetch`, and `web_crawl`. Use `"command": "bun"` on a
Bun-only machine. See `adapters/` for host-specific notes.

## Commands

```
spider-data search <query>     spider-data cache {stats|clear|purge <url>}
spider-data fetch  <url>       spider-data stats [--since 7d]
spider-data crawl  <url>       spider-data doctor
                               spider-data auth  {login|status}
```

Available on all commands: `--json` for machine-readable output, `--refresh` to
bypass the cache, `--debug` to attach a redacted response snippet to errors.

| Command | Flags |
|---|---|
| `search` | `--num <n>` (8), `--country <XX>` (US), `--near "<place>"` (off by default) |
| `fetch` | `--format md\|html\|text`, `--max-chars <n>` (40000), `--section <n>`, `--grep <pattern>`, `--no-escalate`, `--max-paid-rungs <n>` (2), `--render`, `--unblock` |
| `crawl` | `--limit <n>` (20), `--depth <n>` |

`--section` and `--grep` read from the cache and are free. Run
`spider-data --help` for the full list.

## Exit codes

| Code | Meaning |
|---|---|
| 0 | Success, including a search that matched nothing |
| 2 | Usage error |
| 3 | Cross-site redirect, returned rather than followed |
| 4 | Target refuses anonymous access |
| 5 | Target does not exist |
| 6 | Transient |
| 7 | API key rejected |
| 8 | No credit remaining |
| 9 | Rate limited |
| 10 | Refused by local policy (SSRF guard, robots.txt, scheme) |

## Cost tracking

```bash
spider-data stats --since 7d
```

`paid: true` in the audit log means a request left the machine. Cache hits and
free revalidations are never marked paid. The only function that can mark an
attempt paid is the one that opens the socket.

## Security

The API key is never written to `process.env`, never read from a `.env` found by
walking up the directory tree, and never included in an error message. A single
redactor guards all writes to stdout, stderr, the cache, and the log.
`test/leak.test.js` verifies this against a server that reflects request headers
into every response body.

DNS is resolved and each address is checked against the private and reserved
ranges before any request, so the API cannot be used as an SSRF relay. robots.txt
is honoured by default.

## Tests

```bash
npm test
npm run leakcheck
```

184 tests. No network access or API key required.

## Documentation

[`docs/DESIGN.md`](docs/DESIGN.md) covers block detection, the escalation ladder,
and the assumptions the package makes rather than verifies.

## License

MIT. See [LICENSE](LICENSE).