# Agent-Scrape

**Pay-per-call web scraping for AI agents — no API keys, no signup, no account.**

Agents connect over MCP (Model Context Protocol), call `scrape_url()`, get a
`402 Payment Required` challenge, their wallet pays micro-USDC on Base via the
[x402 protocol](https://x402.org), and the scraped markdown comes back.
Payment *is* the auth.

## Live endpoint

**`https://scrape.diazites.online/mcp`** — production MCP endpoint (Base mainnet, x402/USDC).

```
Agent ──tools/call(scrape_url)──▶ Agent-Scrape
        ◀── 402 + PAYMENT-REQUIRED ──
        ── signs USDC payment (wallet) ──▶  Base chain
        ◀────────── 200 + markdown ──────  (verified by facilitator)
```

## Endpoints

| Endpoint | Purpose |
|---|---|
| `POST /mcp` | MCP streamable-HTTP endpoint (initialize / tools/list / tools/call) |
| `GET /pricing` | Agent-readable pricing discovery (payTo, price, network) |
| `GET /healthz` | Health check |
| `GET /` | Landing info |

## Tool pricing

| Tool | Price | Description |
|---|---|---|
| `scrape_url(url, max_chars?)` | $0.01 USDC | Main content of any page as clean markdown |

Network: **Base mainnet** (`eip155:8453`). Assets: **USDC**
(`0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913`).

## Run it

```bash
python3 -m venv venv
venv/bin/pip install -r requirements.txt   # or: mcp x402 fastapi uvicorn trafilatura httpx
cp .env.example .env                        # then fill in keys
venv/bin/python app.py                      # http://0.0.0.0:8093
```

### Configuration (`.env`)

| Variable | Required | Description |
|---|---|---|
| `AGENT_SCRAPE_PAY_TO` | ✅ | Base wallet that receives USDC |
| `AGENT_SCRAPE_CDP_API_KEY_ID` | for payments | Coinbase CDP API key (portal.cdp.coinbase.com) |
| `AGENT_SCRAPE_CDP_API_KEY_SECRET` | for payments | Coinbase CDP API key secret (PEM or Ed25519) |
| `AGENT_SCRAPE_DEV_MODE` | | `1` = bypass payments (demo), `0` = enforce x402 |
| `AGENT_SCRAPE_NETWORK` | | `eip155:8453` mainnet, `eip155:84532` Base Sepolia |
| `AGENT_SCRAPE_PRICE` | | Price in USD per call (default `0.01`) |
| `AGENT_SCRAPE_PORT` | | Default `8093` |

## Seller setup (5 minutes, free)

1. Generate a Base wallet (or use any EVM wallet you control):
   ```bash
   python -c "from eth_account import Account; a=Account.create(); print(a.address, a.key.hex())"
   ```
   Put the address in `AGENT_SCRAPE_PAY_TO`. Fund it with USDC on Base to withdraw (optional — payments can also stay on-chain).
2. Create CDP API keys at https://portal.cdp.coinbase.com → API keys. Paste into `.env`.
3. Set `AGENT_SCRAPE_DEV_MODE=0`, restart. `GET /pricing` now shows production mode.

## How the payment flow works

1. Agent calls `scrape_url` → server responds `402 Payment Required` with a
   `PAYMENT-REQUIRED` header (base64 JSON: scheme `exact`, network
   `eip155:8453`, USDC amount, `payTo` wallet).
2. Agent's x402-capable wallet (Coinbase Smart Wallet, any EIP-3009 signer)
   signs a USDC payment authorization.
3. Server verifies via the Coinbase CDP x402 facilitator (verify → settle on Base).
4. On success the tool executes and the markdown is returned with a
   `PAYMENT-RESPONSE` settlement header.

Dev mode (`AGENT_SCRAPE_DEV_MODE=1`) skips steps 1-3 for testing.

## Try it

```bash
# demo client (dev mode or production)
venv/bin/python e2e_test.py http://localhost:8093/mcp https://example.com

# see the 402 challenge a real agent gets
curl -s -i -X POST http://localhost:8093/mcp \
  -H "Content-Type: application/json" -H "Accept: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"x","version":"1"}}}'
```

## License

MIT. Payments protocol: [x402](https://x402.org) (Apache-2.0).