Skip to main content
Glama
CleanWeb x402 — Smart Web Scraping & YouTube AI Agent

CleanWeb x402 — Smart Web Scraping & YouTube AI Agent

⚡ Multi-Chain & Solana x402 Autonomous AI Agent Suite & Spend Firewall (v2.8.4)

"High-Throughput Clean Web Markdown, YouTube Intelligence & Cryptographic Web3 Oracles for Autonomous AI Agents & Swarms."
Zero credit cards required. Pure B2A (Business-to-Agent) USDC micropayments & pre-funded vaults on Solana, Polygon, Base, and Arbitrum with Solscan Verified IDLs & EIP-712 spend policy attestations.

Version LangChain Official Integration LlamaIndex PyPI npm agentkit npm eliza Tests MCP Multi-Chain License: MIT


🚀 New in v2.8.4 (Community-Driven Release from r/mcp)

CleanWeb v2.8.4 brings major performance & capability upgrades inspired directly by community feedback on r/mcp:

  1. Multimodal Image alt Text Preservation:

    • Preserves high-value image captions and chart descriptions (![alt text](url)) crucial for Multimodal LLMs (Claude 3.5 Sonnet, GPT-4o, Gemini 2.0).

    • Covert Channel Defense: Transparently neutralizes 1x1 tracking pixels and parameter-leaking exfiltration vectors (![alt text][EXFILTRATION_IMAGE_BLOCKED]), ensuring zero data leakage.

  2. Zero-Overhead SPA Hydration State Extractor:

    • Parses client-side Single Page Applications (SPAs) by extracting pre-rendered state from Next.js (__NEXT_DATA__), Nuxt.js (__NUXT_DATA__), and Schema.org JSON-LD (application/ld+json).

    • Pure Python execution with sub-millisecond latency and <16MB RAM—no heavy 500MB headless browser instances required for 90% of dynamic web pages.

    • Optional dynamic_render=True and scroll_depth=N for JavaScript-heavy infinite scroll feeds.


Related MCP server: fetch

📦 Official Multi-Registry Ecosystem Suite (PyPI ↔ npm)

CleanWeb is distributed natively across both Python (PyPI) and TypeScript/Node.js (npm) registries. Install the companion package for your stack—or combine the Python MCP/Oracle Core with TypeScript agent frameworks:

Registry

Package Name

Target Ecosystem

1-Line Install Command

PyPI

x402-cleanweb-agent

Python Core, FastAPI Oracle, Claude/Cursor MCP Server & Dual-Core Eliza Brain

pip install x402-cleanweb-agent (or uvx x402-cleanweb-agent)

PyPI

llama-index-readers-x402-cleanweb

LlamaIndex High-Purity RAG Reader (87% Token Compression & Injection Defense)

pip install llama-index-readers-x402-cleanweb

npm

agentkit-cleanweb

Coinbase AgentKit Action Provider (Base Mainnet EIP-712 & x402 Vaults)

npm install agentkit-cleanweb @coinbase/agentkit

npm

cleanweb-eliza-plugin

ElizaOS v2 Plugin (Solana x402 Micropayments & A.GRID Priority 0 Feed)

npm install cleanweb-eliza-plugin

npm

elizaos-plugin-security-gate

Security Gate & Polygon OMS Spend Firewall (Prompt-Injection Defense)

npm install elizaos-plugin-security-gate

💡 Full-Stack Zero-Trust Pattern (Recommended):

  1. Use cleanweb-eliza-plugin (or x402-cleanweb-agent) for 87% token-compressed, ad-free web ground truth.

  2. Guard outbound transactions and web payloads via elizaos-plugin-security-gate with Polygon Open Money Stack (OMS) spend controls (SecurityGateOMSPolicy.sol verified on Polygon: 0x4A28F9A2E6D0DcB69791980560FC8364A514f205) and sub-second Solana (501) settlements.


🌐 Live Production Gateway & Endpoints


🤖 1-Minute Framework Integration (Coinbase AgentKit, ElizaOS, LangChain & CrewAI)

Equip your autonomous agents across TypeScript (npm) and Python (PyPI) in under 1 minute:

// TypeScript: Official Coinbase AgentKit Action Provider (npm: agentkit-cleanweb)
import { cleanWebActionProvider } from "agentkit-cleanweb";

const cleanWeb = cleanWebActionProvider({ vaultKey: process.env.AGENT_VAULT_KEY });
// TypeScript: Official ElizaOS v2 Plugin (npm: cleanweb-eliza-plugin)
import { cleanWebPlugin } from "cleanweb-eliza-plugin";

export default { name: "AlphaScoutAgent", plugins: [cleanWebPlugin] };
# Python: ElizaOS / Dual-Core Gate-Brain Bridge (PyPI: x402-cleanweb-agent)
from app.integrated_gate_brain_bridge import IntegratedGateBrainBridge

bridge = IntegratedGateBrainBridge(require_x402_payment=False)
res = bridge.execute_pipeline("https://paulgraham.com/greatwork.html", "https://paulgraham.com")
# Python: LlamaIndex High-Purity RAG Reader (pip install llama-index-readers-x402-cleanweb)
from llama_index.readers.x402_cleanweb import X402CleanWebReader
from llama_index.core import VectorStoreIndex

# 87% token-compressed, prompt-injection defended web documents for vector search
documents = X402CleanWebReader().load_data(["https://paulgraham.com/greatwork.html"])
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What are the core habits of great work?")
# Python: LangChain & CrewAI Agent Tools
from examples.langchain_agent_example import clean_web_tool
from examples.crewai_agent_example import clean_web_for_crewai

result = clean_web_tool.invoke({"url": "https://paulgraham.com/greatwork.html"})

⚡ 10-Second Instant Test (Zero Auth / 3 Free Calls Included)

No wallet, credit card, or API key needed. Attach an X-Agent-Nonce header to test immediately:

# Instant JSON Request via curl
curl -X POST "https://x402-cleanweb-agent-7qxtp3324q-du.a.run.app/api/v1/clean-web" \
     -H "Content-Type: application/json" \
     -H "X-Agent-Nonce: instant-trial-uuid-001" \
     -d '{"url": "https://news.ycombinator.com"}'
# Zero-dependency Python snippet
import urllib.request, json

req = urllib.request.Request(
    "https://x402-cleanweb-agent-7qxtp3324q-du.a.run.app/api/v1/clean-web",
    data=json.dumps({"url": "https://news.ycombinator.com"}).encode(),
    headers={"Content-Type": "application/json", "X-Agent-Nonce": "instant-trial-01"}
)
res = json.loads(urllib.request.urlopen(req).read())
print(f"Tokens Saved: {res.get('estimated_token_savings', '87%')}")
print("Clean Markdown Preview:\n", res["clean_markdown"][:300])

💡 Mathematical Token Arbitrage Proof (Economic Rationality)

Autonomous agents route web traffic through x402 because it mathematically saves the operator money on every query:

  • Direct Scraping Cost: 80KB HTML ≈ 20,000 tokens ($0.05 USD input cost on GPT-4o / Claude 3.5 Sonnet).

  • x402 CleanWeb Cost: $0.001 API fee + 2,600 clean tokens ($0.0065 USD) = $0.0075 USD Total.

  • Net Savings: $0.0425 USD saved per web query (85% net dollar reduction).

  • Service Fee ROI: 4,250% ROI on the $0.001 fee.

  • Live Arbitrage Endpoint: GET /api/v1/agent/arbitrage-roi?input_tokens=50000

{
  "status": "economically_optimal",
  "routing_recommendation": "ROUTE_VIA_X402",
  "financial_analysis": {
    "cost_raw_direct_usd": 0.125,
    "cost_with_x402_usd": 0.01725,
    "net_dollar_savings": 0.10775,
    "net_savings_percentage": "86.2%",
    "service_fee_roi_percentage": "10775.0%"
  }
}

🔮 14 Production Agent Tools & Micro-Pricing (USDC)

Tool Name

Endpoint / Function

Cost (USDC)

What It Does

🌐 Clean Web

GET/POST /api/v1/clean-web

0.001 USDC

Strips HTML boilerplate, ads, scripts; returns ad-free Markdown (87% token reduction)

📝 Clean Text

GET /api/v1/clean-text

0.001 USDC

Ultra-lightweight raw plain text for vector/RAG embeddings

🗺️ Site Mapper

GET /api/v1/map-site

0.002 USDC

Recursive sitemap and internal URL tree discovery

🔍 Agent Search

GET /api/v1/search

0.002 USDC

Fast real-time keyword search & verified snippets (Tavily engine)

📑 PDF Extractor

GET /api/v1/clean-pdf

0.005 USDC

Formula- and table-preserved academic paper & research PDF extractor

📦 Batch Clean

POST /api/v1/batch-clean

0.005 USDC

Concurrent parallel scraping for up to 10 URLs in a single request

🎬 YouTube AI

GET /api/v1/clean-youtube

0.010 USDC

Gemini 3.8 Flash multimodal video intelligence & timestamped audio transcript

📊 Extract JSON

POST /api/v1/extract-json

0.030 USDC

Converts arbitrary web content into strict schema-constrained JSON

🔮 Web3 Oracle

POST /api/v1/oracle/grounding

0.035 USDC

Live Search + Clean-to-JSON + EIP-712 Cryptographic On-Chain Attestation

🛡️ Oracle Verify

POST /api/v1/oracle/verify

0.000 USDC

Verifies ECDSA signature of CleanWeb Oracle attestations on-chain

🧠 Deep Research

GET /api/v1/deep-research

0.150 USDC

Multi-source synthesized AI executive research briefing

💼 Vault Deposit

POST /api/v1/vault/deposit

0.000 USDC

Multi-chain USDC vault prefunding for sub-5ms gasless executions

💳 Vault Balance

GET /api/v1/vault/balance

0.000 USDC

Real-time query of credit pass balance and validity

🎫 Pass Status

GET /api/v1/pass-status

0.000 USDC

Inspects status and remaining credits of a prepaid pass


💻 1-Click MCP Setup (Claude Desktop, Cursor, Windsurf)

Connect CleanWeb directly to Claude Desktop, Cursor, or any Model Context Protocol client using uvx:

{
  "mcpServers": {
    "polygon-x402-cleanweb": {
      "command": "uvx",
      "args": ["x402-cleanweb-agent"]
    }
  }
}

Or configure via local clone:

{
  "mcpServers": {
    "polygon-x402-cleanweb": {
      "command": "python",
      "args": ["-u", "mcp_server.py"],
      "env": {
        "POLYGON_RPC_URL": "https://polygon-bor-rpc.publicnode.com",
        "SERVER_WALLET_ADDRESS": "0xA185B43fDD19619f99952AAed6eabf1029bF36a1",
        "USDC_CONTRACT_ADDRESS": "0x3c499c542cEF5E3811e1192ce70d8cC03d5c3359"
      }
    }
  }
}

🤖 Multi-Agent Framework Integrations

Drop-in ready integrations for every major autonomous agent framework via GET /api/v1/agent/integrations/{framework}:

🦜 LangChain / LangGraph

from langchain.tools import tool
import requests

@tool
def clean_web(url: str) -> str:
    """Fetches a URL and returns ad-free, token-optimized Markdown (87% token savings)."""
    res = requests.get(
        "https://x402-cleanweb-agent-7qxtp3324q-du.a.run.app/api/v1/clean-web",
        params={"url": url},
        headers={"X-Agent-Nonce": "langchain-agent-session"}
    )
    return res.json().get("clean_markdown", "")

👥 CrewAI

from crewai.tools import tool
import requests

@tool("CleanWeb Tool")
def clean_web(url: str) -> str:
    """Strips HTML boilerplate and returns pure Markdown to save LLM context window."""
    res = requests.post(
        "https://x402-cleanweb-agent-7qxtp3324q-du.a.run.app/api/v1/clean-web",
        json={"url": url},
        headers={"X-Agent-Nonce": "crewai-agent-session"}
    )
    return res.json().get("clean_markdown", "")

🤖 Microsoft AutoGen

import requests

def clean_web_tool(url: str) -> str:
    res = requests.get(
        "https://x402-cleanweb-agent-7qxtp3324q-du.a.run.app/api/v1/clean-web",
        params={"url": url},
        headers={"X-Agent-Nonce": "autogen-session"}
    )
    return res.json().get("clean_markdown", "")

# assistant.register_for_llm(name="clean_web", description="Clean web markdown")(clean_web_tool)

🏛️ Verified On-Chain Smart Contracts & Accounts (Polygon, Base, Arbitrum, Solana)

CleanWeb operates verified, deterministic smart contracts and native settlement accounts for on-chain EIP-712 oracle verification, autonomous agent pre-funded vault management, and Solana SPL USDC micropayments.

Contract / Account

Polygon Mainnet (137) ✔

Base Mainnet (8453) ✔

Arbitrum One (42161) ✔

Solana Mainnet-Beta (101) ✔

AgentPaymentVault

0x45ecBf...1861

0x28292D...76DD

0x28292D...76DD

7oZ16Y...Wi3y (Solscan)

CleanWebOracleConsumer

0xAECbfB...6D66

0x2394d8...dabe

0x2394d8...dabe

21ZR1Q...bkTL (Solscan)

CleanWebOracleVerifier

0x18fA45...Dc46

0x3eD259...0740

0x3eD259...0740

9nVrym...iopC (Solscan)

Treasury Recipient Wallet

0xA185B4...36a1

0xA185B4...36a1

0xA185B4...36a1

411ksM...9qp (Solscan)

Native USDC Token Tracker

0x3c499c...3359

0x833589...2913

0xaf88d0...5831

EPjFWd...YTDt1v (Solscan)


🧪 Production Readiness Review: 84/84 Tests Passed (100%)

Verified across all 38+ FastAPI routes, 14 agent tools, Dual-Core Gate-Brain cognitive pipeline, multi-chain settlement vaults, and EIP-712 attestations:

python -m pytest tests/ -v

CleanWeb Studio operates strictly under the Transformative Non-Expressive Text/Data Mining (TDM) Fair Use doctrine for autonomous AI reasoning.

  1. No Financial or Investment Warranty: All oracle feeds, search results, and scraped contents are provided AS-IS. CleanWeb Studio assumes zero liability for DeFi smart contract liquidations, prediction market settlements (e.g. Polymarket), or trading losses resulting from external web misreporting or LLM hallucinations.

  2. Caller Responsibility: The autonomous agent deployer/caller is solely responsible for respecting target website intellectual property and applicable laws.

  3. Zero-Data Retention (GDPR Compliant): All fetched HTML and raw assets are processed ephemerally in RAM and purged immediately upon response transmission.

  4. 100% OFAC Sanctions Filtering: All micropayment addresses are verified against OFAC Specially Designated Nationals (SDN) lists.

  5. Full Legal Endpoints:

    • GET /api/v1/legal/terms: Official Machine-to-Machine Terms of Service

    • GET /api/v1/legal/disclaimer: Full Legal & Financial Disclaimer


📜 License

Distributed under the MIT License. See LICENSE for more information.

Available Tools

14 tools
clean_batch_scrapeA

Concurrently scrapes and extracts clean markdown from up to 10 URLs in parallel with high-speed async processing (0.005 USDC).

Usage Guidelines:

  • Use when an agent needs to perform multi-source research across multiple search results simultaneously.

  • Returns: Formatted summary and content preview of parsed web documents.

Inputs: urls (required), auth_token_or_tx. Returns: result

clean_pdf_researchA

Parses and extracts structured plain text, sections, and academic metadata from online PDF whitepapers and research papers.

Usage Guidelines:

  • Use this tool to ingest scientific papers (e.g., arXiv), technical documentation, or financial reports.

  • Constraint: Target document must be a direct HTTP/HTTPS URL pointing to a PDF file under 15MB.

  • Returns: Title, total/parsed page count, word count, and extracted text.

  • Do NOT use for general HTML web pages (use clean_web_content).

  • Do NOT use for YouTube videos (use clean_youtube_transcript).

  • Do NOT use for password-protected, DRM-encrypted, or scanned image-only PDFs without OCR.

Inputs: url (required), max_pages, auth_token_or_tx. Returns: result

clean_text_rawB

Extracts pure, tag-free plain text optimized for RAG embedding and vector indexing (0.001 USDC).

Usage Guidelines:

  • Use this tool when ingesting raw webpage text directly into vector databases (Pinecone, Chroma, Qdrant).

  • Returns: Page title, word count, clean text, and token metrics.

Inputs: url (required), auth_token_or_tx. Returns: result

clean_web_contentA

Scrapes and converts any target web page into clean, LLM-ready structured Markdown, stripping ads, cookie banners, navigation clutter, modals, and script noise.

Usage Guidelines:

  • Use this tool to ingest real-time web articles, blogs, and documentation into LLM context windows.

  • Returns: Clean markdown body, page title, word count, and extraction metadata.

  • Do NOT use for YouTube video parsing (use clean_youtube_transcript).

  • Do NOT use for PDF whitepapers or academic papers (use clean_pdf_research).

  • Do NOT use for paywalled, login-required, or bot-blocked sites.

Inputs: url (required), auth_token_or_tx, respect_robots_txt. Returns: result

clean_youtube_transcriptA

Extracts high-precision subtitles, timestamped transcripts, and comprehensive AI summaries for public YouTube videos using Google Gemini Flash intelligence.

Usage Guidelines:

  • Use this tool to ingest YouTube lecture, tutorial, tech talk, or podcast transcripts into agent workflows.

  • Returns: Video metadata (title, channel, URL), AI Knowledge Summary, and cleaned transcript.

  • Do NOT use for general web pages or articles (use clean_web_content).

  • Do NOT use for PDF documents or papers (use clean_pdf_research).

  • Do NOT use for private, unlisted, age-restricted, or live streams without existing closed captions.

  • If captions are missing or auto-captions fail, the tool reports a detailed fallback error.

Inputs: url (required), lang, auth_token_or_tx. Returns: result

deep_research_topicA

Performs multi-source web crawling and AI synthesis to generate an executive research briefing (0.150 USDC).

Usage Guidelines:

  • Use when an agent needs a comprehensive deep dive into a topic with verified source citations.

  • Returns: Full executive briefing markdown with source citations and key takeaways.

Inputs: query (required), max_sources, auth_token_or_tx. Returns: result

extract_json_schemaA

Extracts schema-constrained structured JSON data from any webpage using Gemini AI (0.030 USDC).

Usage Guidelines:

  • Use when an agent needs structured attributes (e.g. pricing, specs, event dates) directly from a URL.

  • Returns: Clean JSON dictionary matching the requested schema description.

Inputs: url (required), schema_description (required), auth_token_or_tx. Returns: result

get_pass_statusA

Checks the active subscription status, remaining query quota, and validity period for an agent EVM wallet address (0x...) or Agent VIP Pass Token.

Usage Guidelines:

  • Use to verify micropayment allowance or query entitlements before dispatching heavy scrape batches.

  • Returns: Pass tier, remaining balance, and expiration timestamp.

Inputs: agent_wallet_or_token (required). Returns: result

get_payment_infoA

Returns complete Web3 x402 micropayment configuration, supported multi-chain USDC contract addresses, EVM network chain IDs (Polygon: 137, Base: 8453, Arbitrum: 42161), recipient wallet address, and pricing tiers for all CleanWeb Studio agent tools.

Usage Guidelines:

  • Use this tool to discover network parameters and deposit requirements before making x402 paid queries.

  • Returns: Structured pricing markdown table, contract addresses, and pre-funded vault endpoints.

  • Do NOT use for checking individual wallet balances (use get_vault_balance).

Inputs: tier, chain. Returns: result

get_vault_balanceA

Checks the remaining pre-funded USDC balance, total usage, and session status for an agent wallet address or session key.

Usage Guidelines:

  • Use this tool before executing heavy tasks to verify sufficient balance for zero-latency execution.

  • Returns: Agent address, available balance in USDC, total deposited, total consumed, queries handled, and session key.

  • Do NOT use for querying pricing or chain parameters (use get_payment_info).

Inputs: agent_address_or_key (required). Returns: result

map_siteA

Discovers domain sitemap or traverses internal anchor links to return a canonical URL tree (0.002 USDC).

Usage Guidelines:

  • Use this tool to map an entire website or documentation site before scraping.

  • Firecrawl /map equivalent for autonomous web navigation agents.

  • Returns: Target URL, domain, total URL count, list of discovered URLs, and sitemap detection status.

Inputs: url (required), max_links, auth_token_or_tx. Returns: result

oracle_groundingA

Executes real-time web search, noise-free markdown extraction, Gemini AI JSON structuring, and cryptographically signs the result with an on-chain verifiable EIP-712 attestation (0.035 USDC).

Usage Guidelines:

  • Use this tool when an autonomous agent or smart contract requires verified, tamper-proof ground truth from the live web.

  • Returns: Structured facts JSON, human/LLM readable summary markdown, source URLs, and EIP-712 cryptographic signature (v, r, s).

  • Smart contracts can verify this off-chain or on-chain using CleanWebOracleVerifier.sol.

Inputs: query (required), max_sources, auth_token_or_tx, target_schema_json. Returns: result

search_web_quickB

Performs fast real-time keyword web search returning titles, links, and text snippets (0.002 USDC).

Usage Guidelines:

  • Tavily / s.jina.ai competitor designed specifically for LLM autonomous agent retrieval.

  • Returns: Concise search result list with title, verified URL, and text snippet.

Inputs: query (required), max_results, auth_token_or_tx. Returns: result

verify_oracle_attestationA

Verifies an EIP-712 cryptographic attestation produced by CleanWeb Oracle off-chain without consuming gas.

Usage Guidelines:

  • Use this tool to mathematically verify that data received from CleanWeb Oracle has not been tampered with.

  • Returns: Verification status (valid/invalid), recovered signer address, expected oracle address, and message.

Inputs: query (required), data_hash (required), signature (required), timestamp (required). Returns: result

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.9/5.0

    Scored across 14 tools

    Disambiguation4/5

    Most tools have clearly distinct purposes, and descriptions explicitly cross-reference alternatives (e.g., clean_web_content vs clean_youtube_transcript vs clean_pdf_research). Minor ambiguity remains between clean_text_raw and clean_web_content, and between search_web_quick, deep_research_topic, and oracle_grounding, though the latter are differentiated by cost, depth, and attestation.

    Naming Consistency4/5

    All names are snake_case and generally follow verb_noun patterns (get_*, clean_*, verify_*, map_*, search_*, extract_*). However, oracle_grounding and deep_research_topic break the verb-first pattern, and clean_* is used both for scraping and a text extraction variant.

    Tool Count5/5

    14 tools is well within the ideal 3-15 range and each tool appears to earn its place by covering a distinct scraping, extraction, search, payment, or verification capability. The count feels scoped to the server's purpose without obvious padding.

    Completeness4/5

    The surface covers web scraping, YouTube transcripts, PDFs, batch scraping, search, sitemap mapping, structured extraction, deep research, oracle attestation, and payment/status checks. Minor gaps exist, such as no explicit tool for managing or depositing funds, but core agent workflows are well supported.

    Maintenance

    ActivityActive
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      B
      maintenance
      Pay-per-use clean web reader for AI agents. URL in, markdown plus metadata out, in milliseconds. Settled per-call in USDC over x402 — no signup, no API keys.
      1
      102 npm
      3
      MIT
    • A
      license
      Not graded
      quality
      D
      maintenance
      Fetches web pages and converts them to markdown for LLM consumption, supporting chunked reading and raw content extraction.
      MIT
    • A
      license
      Not graded
      quality
      D
      maintenance
      MCP server for headless browser automation using Puppeteer, enabling AI to navigate, click, fill forms, take screenshots, and execute JavaScript on web pages.
      MIT
    • A
      license
      Not graded
      quality
      D
      maintenance
      MCP server providing 17 keyless, pay-per-use web-data tools with signed-provenance receipts, enabling AI agents to autonomously fetch, extract, and verify web content on Base mainnet.
      48 npm
      MIT