Agent-Scrape

Pay-per-call web scraping for AI agents — no API keys, no signup, no account.

Agents connect over MCP (Model Context Protocol), call scrape_url(), get a 402 Payment Required challenge, their wallet pays micro-USDC on Base via the x402 protocol, and the scraped markdown comes back. Payment is the auth.

Live endpoint

https://scrape.diazites.online/mcp — production MCP endpoint (Base mainnet, x402/USDC).

Agent ──tools/call(scrape_url)──▶ Agent-Scrape
        ◀── 402 + PAYMENT-REQUIRED ──
        ── signs USDC payment (wallet) ──▶  Base chain
        ◀────────── 200 + markdown ──────  (verified by facilitator)

Related MCP server: mdapi-mcp

Endpoints

Tool pricing

Network: Base mainnet (eip155:8453). Assets: USDC (0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913).

Run it

python3 -m venv venv
venv/bin/pip install -r requirements.txt   # or: mcp x402 fastapi uvicorn trafilatura httpx
cp .env.example .env                        # then fill in keys
venv/bin/python app.py                      # http://0.0.0.0:8093

Configuration (.env)

Seller setup (5 minutes, free)

1. Generate a Base wallet (or use any EVM wallet you control):

   python -c "from eth_account import Account; a=Account.create(); print(a.address, a.key.hex())"

   Put the address in AGENT_SCRAPE_PAY_TO. Fund it with USDC on Base to withdraw (optional — payments can also stay on-chain).

2. Create CDP API keys at https://portal.cdp.coinbase.com → API keys. Paste into .env.

3. Set AGENT_SCRAPE_DEV_MODE=0, restart. GET /pricing now shows production mode.

How the payment flow works

1. Agent calls scrape_url → server responds 402 Payment Required with a PAYMENT-REQUIRED header (base64 JSON: scheme exact, network eip155:8453, USDC amount, payTo wallet).

2. Agent's x402-capable wallet (Coinbase Smart Wallet, any EIP-3009 signer) signs a USDC payment authorization.

3. Server verifies via the Coinbase CDP x402 facilitator (verify → settle on Base).

4. On success the tool executes and the markdown is returned with a PAYMENT-RESPONSE settlement header.

Dev mode (AGENT_SCRAPE_DEV_MODE=1) skips steps 1-3 for testing.

Try it

# demo client (dev mode or production)
venv/bin/python e2e_test.py http://localhost:8093/mcp https://example.com

# see the 402 challenge a real agent gets
curl -s -i -X POST http://localhost:8093/mcp \
  -H "Content-Type: application/json" -H "Accept: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"x","version":"1"}}}'

License

MIT. Payments protocol: x402 (Apache-2.0).