Skip to main content
Glama

Agent-Scrape

Pay-per-call web scraping for AI agents — no API keys, no signup, no account.

Agents connect over MCP (Model Context Protocol), call scrape_url(), get a 402 Payment Required challenge, their wallet pays micro-USDC on Base via the x402 protocol, and the scraped markdown comes back. Payment is the auth.

Live endpoint

https://scrape.diazites.online/mcp — production MCP endpoint (Base mainnet, x402/USDC).

Agent ──tools/call(scrape_url)──▶ Agent-Scrape
        ◀── 402 + PAYMENT-REQUIRED ──
        ── signs USDC payment (wallet) ──▶  Base chain
        ◀────────── 200 + markdown ──────  (verified by facilitator)

Related MCP server: mdapi-mcp

Endpoints

Endpoint

Purpose

POST /mcp

MCP streamable-HTTP endpoint (initialize / tools/list / tools/call)

GET /pricing

Agent-readable pricing discovery (payTo, price, network)

GET /healthz

Health check

GET /

Landing info

Tool pricing

Tool

Price

Description

scrape_url(url, max_chars?)

$0.01 USDC

Main content of any page as clean markdown

Network: Base mainnet (eip155:8453). Assets: USDC (0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913).

Run it

python3 -m venv venv
venv/bin/pip install -r requirements.txt   # or: mcp x402 fastapi uvicorn trafilatura httpx
cp .env.example .env                        # then fill in keys
venv/bin/python app.py                      # http://0.0.0.0:8093

Configuration (.env)

Variable

Required

Description

AGENT_SCRAPE_PAY_TO

Base wallet that receives USDC

AGENT_SCRAPE_CDP_API_KEY_ID

for payments

Coinbase CDP API key (portal.cdp.coinbase.com)

AGENT_SCRAPE_CDP_API_KEY_SECRET

for payments

Coinbase CDP API key secret (PEM or Ed25519)

AGENT_SCRAPE_DEV_MODE

1 = bypass payments (demo), 0 = enforce x402

AGENT_SCRAPE_NETWORK

eip155:8453 mainnet, eip155:84532 Base Sepolia

AGENT_SCRAPE_PRICE

Price in USD per call (default 0.01)

AGENT_SCRAPE_PORT

Default 8093

Seller setup (5 minutes, free)

  1. Generate a Base wallet (or use any EVM wallet you control):

    python -c "from eth_account import Account; a=Account.create(); print(a.address, a.key.hex())"

    Put the address in AGENT_SCRAPE_PAY_TO. Fund it with USDC on Base to withdraw (optional — payments can also stay on-chain).

  2. Create CDP API keys at https://portal.cdp.coinbase.com → API keys. Paste into .env.

  3. Set AGENT_SCRAPE_DEV_MODE=0, restart. GET /pricing now shows production mode.

How the payment flow works

  1. Agent calls scrape_url → server responds 402 Payment Required with a PAYMENT-REQUIRED header (base64 JSON: scheme exact, network eip155:8453, USDC amount, payTo wallet).

  2. Agent's x402-capable wallet (Coinbase Smart Wallet, any EIP-3009 signer) signs a USDC payment authorization.

  3. Server verifies via the Coinbase CDP x402 facilitator (verify → settle on Base).

  4. On success the tool executes and the markdown is returned with a PAYMENT-RESPONSE settlement header.

Dev mode (AGENT_SCRAPE_DEV_MODE=1) skips steps 1-3 for testing.

Try it

# demo client (dev mode or production)
venv/bin/python e2e_test.py http://localhost:8093/mcp https://example.com

# see the 402 challenge a real agent gets
curl -s -i -X POST http://localhost:8093/mcp \
  -H "Content-Type: application/json" -H "Accept: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"x","version":"1"}}}'

License

MIT. Payments protocol: x402 (Apache-2.0).

Related MCP Connectors

Related MCP Servers

  • A
    license
    A
    quality
    A
    maintenance
    Context-efficient MCP server for AI agents. fetch_extract reduces token usage by a median 98.1% (53,820 → 2,001 tokens) vs raw HTML — saves ~$0.156/call at Sonnet pricing. Pay $0.02 USDC on Base via x402. First call free per wallet. 10 always-free utility tools included.
    33
    1
    MIT
  • A
    license
    Not graded
    quality
    C
    maintenance
    MCP server for converting HTML to Markdown and extracting page data, with pay-per-call using USDC on Base.
    MIT
  • A
    license
    A
    quality
    B
    maintenance
    MCP server that enables AI agents to fetch web content as clean, structured Markdown via NanoParse's hosted API, using x402 micropayments in USDC on Base.
    1
    34 npm
    MIT