CleanWeb x402 — Smart Web Scraping & YouTube AI Agent
This server provides x402-paid web scraping, content extraction, AI research, and Web3 oracle/vault utilities for autonomous agents.
Discover multi-chain USDC payment config, contract addresses, chain IDs, recipient wallets, and tool pricing with
get_payment_info.Scrape general webpages into clean LLM-ready Markdown with
clean_web_content.Extract timestamped YouTube transcripts and AI summaries with
clean_youtube_transcript.Parse online PDF research papers and whitepapers with
clean_pdf_research.Check prepaid USDC vault balance, usage, and session status with
get_vault_balance.Generate live-web grounded facts, structured JSON, and EIP-712 cryptographic attestations with
oracle_grounding.Verify oracle attestations off-chain without gas using
verify_oracle_attestation.Extract raw plain text optimized for RAG/vector databases with
clean_text_raw.Discover sitemaps and internal URL trees with
map_site.Run fast real-time keyword web searches with
search_web_quick.Extract schema-constrained JSON from any webpage with
extract_json_schema.Generate multi-source executive research briefings with
deep_research_topic.Scrape and extract clean Markdown from up to 10 URLs in parallel with
clean_batch_scrape.Check subscription/quota status for agent wallets or VIP pass tokens with
get_pass_status.
⚡ Multi-Chain & Solana x402 Autonomous AI Agent Suite & Spend Firewall (v2.8.4)
"High-Throughput Clean Web Markdown, YouTube Intelligence & Cryptographic Web3 Oracles for Autonomous AI Agents & Swarms."
Zero credit cards required. Pure B2A (Business-to-Agent) USDC micropayments & pre-funded vaults on Solana, Polygon, Base, and Arbitrum with Solscan Verified IDLs & EIP-712 spend policy attestations.
🚀 New in v2.8.4 (Community-Driven Release from r/mcp)
CleanWeb v2.8.4 brings major performance & capability upgrades inspired directly by community feedback on r/mcp:
Multimodal Image
altText Preservation:Preserves high-value image captions and chart descriptions (
) crucial for Multimodal LLMs (Claude 3.5 Sonnet, GPT-4o, Gemini 2.0).Covert Channel Defense: Transparently neutralizes 1x1 tracking pixels and parameter-leaking exfiltration vectors (
![alt text][EXFILTRATION_IMAGE_BLOCKED]), ensuring zero data leakage.
Zero-Overhead SPA Hydration State Extractor:
Parses client-side Single Page Applications (SPAs) by extracting pre-rendered state from Next.js (
__NEXT_DATA__), Nuxt.js (__NUXT_DATA__), and Schema.org JSON-LD (application/ld+json).Pure Python execution with sub-millisecond latency and <16MB RAM—no heavy 500MB headless browser instances required for 90% of dynamic web pages.
Optional
dynamic_render=Trueandscroll_depth=Nfor JavaScript-heavy infinite scroll feeds.
Related MCP server: fetch
📦 Official Multi-Registry Ecosystem Suite (PyPI ↔ npm)
CleanWeb is distributed natively across both Python (PyPI) and TypeScript/Node.js (npm) registries. Install the companion package for your stack—or combine the Python MCP/Oracle Core with TypeScript agent frameworks:
Registry | Package Name | Target Ecosystem | 1-Line Install Command |
PyPI | Python Core, FastAPI Oracle, Claude/Cursor MCP Server & Dual-Core Eliza Brain |
| |
PyPI | LlamaIndex High-Purity RAG Reader (87% Token Compression & Injection Defense) |
| |
npm | Coinbase AgentKit Action Provider (Base Mainnet EIP-712 & x402 Vaults) |
| |
npm | ElizaOS v2 Plugin (Solana x402 Micropayments & A.GRID Priority 0 Feed) |
| |
npm | Security Gate & Polygon OMS Spend Firewall (Prompt-Injection Defense) |
|
💡 Full-Stack Zero-Trust Pattern (Recommended):
Use
cleanweb-eliza-plugin(orx402-cleanweb-agent) for 87% token-compressed, ad-free web ground truth.Guard outbound transactions and web payloads via
elizaos-plugin-security-gatewith Polygon Open Money Stack (OMS) spend controls (SecurityGateOMSPolicy.solverified on Polygon:0x4A28F9A2E6D0DcB69791980560FC8364A514f205) and sub-second Solana (501) settlements.
🌐 Live Production Gateway & Endpoints
Production Gateway:
https://x402-cleanweb-agent-7qxtp3324q-du.a.run.appAgent LLM Standard (
llms.txt):https://x402-cleanweb-agent-7qxtp3324q-du.a.run.app/llms.txtA2A Agent Manifest:
/.well-known/agent.jsonOpenAI Plugin Manifest:
/.well-known/ai-plugin.jsonMCP Server Card:
/.well-known/mcp/server-card.jsonArbitrage ROI Calculator:
/api/v1/agent/arbitrage-roiInteractive Studio Dashboard:
https://x402-cleanweb-agent-7qxtp3324q-du.a.run.app/dashboardInteractive OpenAPI Specs:
https://x402-cleanweb-agent-7qxtp3324q-du.a.run.app/docs
🤖 1-Minute Framework Integration (Coinbase AgentKit, ElizaOS, LangChain & CrewAI)
Equip your autonomous agents across TypeScript (npm) and Python (PyPI) in under 1 minute:
// TypeScript: Official Coinbase AgentKit Action Provider (npm: agentkit-cleanweb)
import { cleanWebActionProvider } from "agentkit-cleanweb";
const cleanWeb = cleanWebActionProvider({ vaultKey: process.env.AGENT_VAULT_KEY });// TypeScript: Official ElizaOS v2 Plugin (npm: cleanweb-eliza-plugin)
import { cleanWebPlugin } from "cleanweb-eliza-plugin";
export default { name: "AlphaScoutAgent", plugins: [cleanWebPlugin] };# Python: ElizaOS / Dual-Core Gate-Brain Bridge (PyPI: x402-cleanweb-agent)
from app.integrated_gate_brain_bridge import IntegratedGateBrainBridge
bridge = IntegratedGateBrainBridge(require_x402_payment=False)
res = bridge.execute_pipeline("https://paulgraham.com/greatwork.html", "https://paulgraham.com")# Python: LlamaIndex High-Purity RAG Reader (pip install llama-index-readers-x402-cleanweb)
from llama_index.readers.x402_cleanweb import X402CleanWebReader
from llama_index.core import VectorStoreIndex
# 87% token-compressed, prompt-injection defended web documents for vector search
documents = X402CleanWebReader().load_data(["https://paulgraham.com/greatwork.html"])
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What are the core habits of great work?")# Python: LangChain & CrewAI Agent Tools
from examples.langchain_agent_example import clean_web_tool
from examples.crewai_agent_example import clean_web_for_crewai
result = clean_web_tool.invoke({"url": "https://paulgraham.com/greatwork.html"})⚡ 10-Second Instant Test (Zero Auth / 3 Free Calls Included)
No wallet, credit card, or API key needed. Attach an X-Agent-Nonce header to test immediately:
# Instant JSON Request via curl
curl -X POST "https://x402-cleanweb-agent-7qxtp3324q-du.a.run.app/api/v1/clean-web" \
-H "Content-Type: application/json" \
-H "X-Agent-Nonce: instant-trial-uuid-001" \
-d '{"url": "https://news.ycombinator.com"}'# Zero-dependency Python snippet
import urllib.request, json
req = urllib.request.Request(
"https://x402-cleanweb-agent-7qxtp3324q-du.a.run.app/api/v1/clean-web",
data=json.dumps({"url": "https://news.ycombinator.com"}).encode(),
headers={"Content-Type": "application/json", "X-Agent-Nonce": "instant-trial-01"}
)
res = json.loads(urllib.request.urlopen(req).read())
print(f"Tokens Saved: {res.get('estimated_token_savings', '87%')}")
print("Clean Markdown Preview:\n", res["clean_markdown"][:300])💡 Mathematical Token Arbitrage Proof (Economic Rationality)
Autonomous agents route web traffic through x402 because it mathematically saves the operator money on every query:
Direct Scraping Cost: 80KB HTML ≈ 20,000 tokens ($0.05 USD input cost on GPT-4o / Claude 3.5 Sonnet).
x402 CleanWeb Cost: $0.001 API fee + 2,600 clean tokens ($0.0065 USD) = $0.0075 USD Total.
Net Savings: $0.0425 USD saved per web query (85% net dollar reduction).
Service Fee ROI: 4,250% ROI on the $0.001 fee.
Live Arbitrage Endpoint:
GET /api/v1/agent/arbitrage-roi?input_tokens=50000
{
"status": "economically_optimal",
"routing_recommendation": "ROUTE_VIA_X402",
"financial_analysis": {
"cost_raw_direct_usd": 0.125,
"cost_with_x402_usd": 0.01725,
"net_dollar_savings": 0.10775,
"net_savings_percentage": "86.2%",
"service_fee_roi_percentage": "10775.0%"
}
}🔮 14 Production Agent Tools & Micro-Pricing (USDC)
Tool Name | Endpoint / Function | Cost (USDC) | What It Does |
🌐 Clean Web |
| 0.001 USDC | Strips HTML boilerplate, ads, scripts; returns ad-free Markdown (87% token reduction) |
📝 Clean Text |
| 0.001 USDC | Ultra-lightweight raw plain text for vector/RAG embeddings |
🗺️ Site Mapper |
| 0.002 USDC | Recursive sitemap and internal URL tree discovery |
🔍 Agent Search |
| 0.002 USDC | Fast real-time keyword search & verified snippets (Tavily engine) |
📑 PDF Extractor |
| 0.005 USDC | Formula- and table-preserved academic paper & research PDF extractor |
📦 Batch Clean |
| 0.005 USDC | Concurrent parallel scraping for up to 10 URLs in a single request |
🎬 YouTube AI |
| 0.010 USDC | Gemini 3.8 Flash multimodal video intelligence & timestamped audio transcript |
📊 Extract JSON |
| 0.030 USDC | Converts arbitrary web content into strict schema-constrained JSON |
🔮 Web3 Oracle |
| 0.035 USDC | Live Search + Clean-to-JSON + EIP-712 Cryptographic On-Chain Attestation |
🛡️ Oracle Verify |
| 0.000 USDC | Verifies ECDSA signature of CleanWeb Oracle attestations on-chain |
🧠 Deep Research |
| 0.150 USDC | Multi-source synthesized AI executive research briefing |
💼 Vault Deposit |
| 0.000 USDC | Multi-chain USDC vault prefunding for sub-5ms gasless executions |
💳 Vault Balance |
| 0.000 USDC | Real-time query of credit pass balance and validity |
🎫 Pass Status |
| 0.000 USDC | Inspects status and remaining credits of a prepaid pass |
💻 1-Click MCP Setup (Claude Desktop, Cursor, Windsurf)
Connect CleanWeb directly to Claude Desktop, Cursor, or any Model Context Protocol client using uvx:
{
"mcpServers": {
"polygon-x402-cleanweb": {
"command": "uvx",
"args": ["x402-cleanweb-agent"]
}
}
}Or configure via local clone:
{
"mcpServers": {
"polygon-x402-cleanweb": {
"command": "python",
"args": ["-u", "mcp_server.py"],
"env": {
"POLYGON_RPC_URL": "https://polygon-bor-rpc.publicnode.com",
"SERVER_WALLET_ADDRESS": "0xA185B43fDD19619f99952AAed6eabf1029bF36a1",
"USDC_CONTRACT_ADDRESS": "0x3c499c542cEF5E3811e1192ce70d8cC03d5c3359"
}
}
}
}🤖 Multi-Agent Framework Integrations
Drop-in ready integrations for every major autonomous agent framework via GET /api/v1/agent/integrations/{framework}:
🦜 LangChain / LangGraph
from langchain.tools import tool
import requests
@tool
def clean_web(url: str) -> str:
"""Fetches a URL and returns ad-free, token-optimized Markdown (87% token savings)."""
res = requests.get(
"https://x402-cleanweb-agent-7qxtp3324q-du.a.run.app/api/v1/clean-web",
params={"url": url},
headers={"X-Agent-Nonce": "langchain-agent-session"}
)
return res.json().get("clean_markdown", "")👥 CrewAI
from crewai.tools import tool
import requests
@tool("CleanWeb Tool")
def clean_web(url: str) -> str:
"""Strips HTML boilerplate and returns pure Markdown to save LLM context window."""
res = requests.post(
"https://x402-cleanweb-agent-7qxtp3324q-du.a.run.app/api/v1/clean-web",
json={"url": url},
headers={"X-Agent-Nonce": "crewai-agent-session"}
)
return res.json().get("clean_markdown", "")🤖 Microsoft AutoGen
import requests
def clean_web_tool(url: str) -> str:
res = requests.get(
"https://x402-cleanweb-agent-7qxtp3324q-du.a.run.app/api/v1/clean-web",
params={"url": url},
headers={"X-Agent-Nonce": "autogen-session"}
)
return res.json().get("clean_markdown", "")
# assistant.register_for_llm(name="clean_web", description="Clean web markdown")(clean_web_tool)🏛️ Verified On-Chain Smart Contracts & Accounts (Polygon, Base, Arbitrum, Solana)
CleanWeb operates verified, deterministic smart contracts and native settlement accounts for on-chain EIP-712 oracle verification, autonomous agent pre-funded vault management, and Solana SPL USDC micropayments.
Contract / Account | Polygon Mainnet (137) ✔ | Base Mainnet (8453) ✔ | Arbitrum One (42161) ✔ | Solana Mainnet-Beta (101) ✔ |
| ||||
| ||||
| ||||
| ||||
|
🧪 Production Readiness Review: 84/84 Tests Passed (100%)
Verified across all 38+ FastAPI routes, 14 agent tools, Dual-Core Gate-Brain cognitive pipeline, multi-chain settlement vaults, and EIP-712 attestations:
python -m pytest tests/ -v⚖️ Legal, Fair Use & Financial Non-Liability Disclaimer
CleanWeb Studio operates strictly under the Transformative Non-Expressive Text/Data Mining (TDM) Fair Use doctrine for autonomous AI reasoning.
No Financial or Investment Warranty: All oracle feeds, search results, and scraped contents are provided AS-IS. CleanWeb Studio assumes zero liability for DeFi smart contract liquidations, prediction market settlements (e.g. Polymarket), or trading losses resulting from external web misreporting or LLM hallucinations.
Caller Responsibility: The autonomous agent deployer/caller is solely responsible for respecting target website intellectual property and applicable laws.
Zero-Data Retention (GDPR Compliant): All fetched HTML and raw assets are processed ephemerally in RAM and purged immediately upon response transmission.
100% OFAC Sanctions Filtering: All micropayment addresses are verified against OFAC Specially Designated Nationals (SDN) lists.
Full Legal Endpoints:
GET /api/v1/legal/terms: Official Machine-to-Machine Terms of ServiceGET /api/v1/legal/disclaimer: Full Legal & Financial Disclaimer
📜 License
Distributed under the MIT License. See LICENSE for more information.
Available Tools
14 toolsclean_batch_scrapeA
Concurrently scrapes and extracts clean markdown from up to 10 URLs in parallel with high-speed async processing (0.005 USDC).
Usage Guidelines:
Use when an agent needs to perform multi-source research across multiple search results simultaneously.
Returns: Formatted summary and content preview of parsed web documents.
Inputs: urls (required), auth_token_or_tx. Returns: result
clean_pdf_researchA
Parses and extracts structured plain text, sections, and academic metadata from online PDF whitepapers and research papers.
Usage Guidelines:
Use this tool to ingest scientific papers (e.g., arXiv), technical documentation, or financial reports.
Constraint: Target document must be a direct HTTP/HTTPS URL pointing to a PDF file under 15MB.
Returns: Title, total/parsed page count, word count, and extracted text.
Do NOT use for general HTML web pages (use
clean_web_content).Do NOT use for YouTube videos (use
clean_youtube_transcript).Do NOT use for password-protected, DRM-encrypted, or scanned image-only PDFs without OCR.
Inputs: url (required), max_pages, auth_token_or_tx. Returns: result
clean_text_rawB
Extracts pure, tag-free plain text optimized for RAG embedding and vector indexing (0.001 USDC).
Usage Guidelines:
Use this tool when ingesting raw webpage text directly into vector databases (Pinecone, Chroma, Qdrant).
Returns: Page title, word count, clean text, and token metrics.
Inputs: url (required), auth_token_or_tx. Returns: result
clean_web_contentA
Scrapes and converts any target web page into clean, LLM-ready structured Markdown, stripping ads, cookie banners, navigation clutter, modals, and script noise.
Usage Guidelines:
Use this tool to ingest real-time web articles, blogs, and documentation into LLM context windows.
Returns: Clean markdown body, page title, word count, and extraction metadata.
Do NOT use for YouTube video parsing (use
clean_youtube_transcript).Do NOT use for PDF whitepapers or academic papers (use
clean_pdf_research).Do NOT use for paywalled, login-required, or bot-blocked sites.
Inputs: url (required), auth_token_or_tx, respect_robots_txt. Returns: result
clean_youtube_transcriptA
Extracts high-precision subtitles, timestamped transcripts, and comprehensive AI summaries for public YouTube videos using Google Gemini Flash intelligence.
Usage Guidelines:
Use this tool to ingest YouTube lecture, tutorial, tech talk, or podcast transcripts into agent workflows.
Returns: Video metadata (title, channel, URL), AI Knowledge Summary, and cleaned transcript.
Do NOT use for general web pages or articles (use
clean_web_content).Do NOT use for PDF documents or papers (use
clean_pdf_research).Do NOT use for private, unlisted, age-restricted, or live streams without existing closed captions.
If captions are missing or auto-captions fail, the tool reports a detailed fallback error.
Inputs: url (required), lang, auth_token_or_tx. Returns: result
deep_research_topicA
Performs multi-source web crawling and AI synthesis to generate an executive research briefing (0.150 USDC).
Usage Guidelines:
Use when an agent needs a comprehensive deep dive into a topic with verified source citations.
Returns: Full executive briefing markdown with source citations and key takeaways.
Inputs: query (required), max_sources, auth_token_or_tx. Returns: result
extract_json_schemaA
Extracts schema-constrained structured JSON data from any webpage using Gemini AI (0.030 USDC).
Usage Guidelines:
Use when an agent needs structured attributes (e.g. pricing, specs, event dates) directly from a URL.
Returns: Clean JSON dictionary matching the requested schema description.
Inputs: url (required), schema_description (required), auth_token_or_tx. Returns: result
get_pass_statusA
Checks the active subscription status, remaining query quota, and validity period for an agent EVM wallet address (0x...) or Agent VIP Pass Token.
Usage Guidelines:
Use to verify micropayment allowance or query entitlements before dispatching heavy scrape batches.
Returns: Pass tier, remaining balance, and expiration timestamp.
Inputs: agent_wallet_or_token (required). Returns: result
get_payment_infoA
Returns complete Web3 x402 micropayment configuration, supported multi-chain USDC contract addresses, EVM network chain IDs (Polygon: 137, Base: 8453, Arbitrum: 42161), recipient wallet address, and pricing tiers for all CleanWeb Studio agent tools.
Usage Guidelines:
Use this tool to discover network parameters and deposit requirements before making x402 paid queries.
Returns: Structured pricing markdown table, contract addresses, and pre-funded vault endpoints.
Do NOT use for checking individual wallet balances (use
get_vault_balance).
Inputs: tier, chain. Returns: result
get_vault_balanceA
Checks the remaining pre-funded USDC balance, total usage, and session status for an agent wallet address or session key.
Usage Guidelines:
Use this tool before executing heavy tasks to verify sufficient balance for zero-latency execution.
Returns: Agent address, available balance in USDC, total deposited, total consumed, queries handled, and session key.
Do NOT use for querying pricing or chain parameters (use
get_payment_info).
Inputs: agent_address_or_key (required). Returns: result
map_siteA
Discovers domain sitemap or traverses internal anchor links to return a canonical URL tree (0.002 USDC).
Usage Guidelines:
Use this tool to map an entire website or documentation site before scraping.
Firecrawl /map equivalent for autonomous web navigation agents.
Returns: Target URL, domain, total URL count, list of discovered URLs, and sitemap detection status.
Inputs: url (required), max_links, auth_token_or_tx. Returns: result
oracle_groundingA
Executes real-time web search, noise-free markdown extraction, Gemini AI JSON structuring, and cryptographically signs the result with an on-chain verifiable EIP-712 attestation (0.035 USDC).
Usage Guidelines:
Use this tool when an autonomous agent or smart contract requires verified, tamper-proof ground truth from the live web.
Returns: Structured facts JSON, human/LLM readable summary markdown, source URLs, and EIP-712 cryptographic signature (v, r, s).
Smart contracts can verify this off-chain or on-chain using CleanWebOracleVerifier.sol.
Inputs: query (required), max_sources, auth_token_or_tx, target_schema_json. Returns: result
search_web_quickB
Performs fast real-time keyword web search returning titles, links, and text snippets (0.002 USDC).
Usage Guidelines:
Tavily / s.jina.ai competitor designed specifically for LLM autonomous agent retrieval.
Returns: Concise search result list with title, verified URL, and text snippet.
Inputs: query (required), max_results, auth_token_or_tx. Returns: result
verify_oracle_attestationA
Verifies an EIP-712 cryptographic attestation produced by CleanWeb Oracle off-chain without consuming gas.
Usage Guidelines:
Use this tool to mathematically verify that data received from CleanWeb Oracle has not been tampered with.
Returns: Verification status (valid/invalid), recovered signer address, expected oracle address, and message.
Inputs: query (required), data_hash (required), signature (required), timestamp (required). Returns: result
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 14 tools
Most tools have clearly distinct purposes, and descriptions explicitly cross-reference alternatives (e.g., clean_web_content vs clean_youtube_transcript vs clean_pdf_research). Minor ambiguity remains between clean_text_raw and clean_web_content, and between search_web_quick, deep_research_topic, and oracle_grounding, though the latter are differentiated by cost, depth, and attestation.
All names are snake_case and generally follow verb_noun patterns (get_*, clean_*, verify_*, map_*, search_*, extract_*). However, oracle_grounding and deep_research_topic break the verb-first pattern, and clean_* is used both for scraping and a text extraction variant.
14 tools is well within the ideal 3-15 range and each tool appears to earn its place by covering a distinct scraping, extraction, search, payment, or verification capability. The count feels scoped to the server's purpose without obvious padding.
The surface covers web scraping, YouTube transcripts, PDFs, batch scraping, search, sitemap mapping, structured extraction, deep research, oracle attestation, and payment/status checks. Minor gaps exist, such as no explicit tool for managing or depositing funds, but core agent workflows are well supported.
Maintenance
Related MCP Connectors
Stealth scraping API for AI agents. Clean Markdown from any URL. x402 crypto payments.
Pay-per-request webpage-to-Markdown extraction for AI agents. $0.005 USDC via x402 on Solana.
Pay-per-call (x402/USDC-Base) web + crypto data tools for AI agents: audit, extract, crypto, DeFi.
Pay-per-call web reader (Markdown/RAG chunks) and agent utilities; x402 USDC on Base.
Related MCP Servers
- AlicenseAqualityBmaintenancePay-per-use clean web reader for AI agents. URL in, markdown plus metadata out, in milliseconds. Settled per-call in USDC over x402 — no signup, no API keys.1102 npm3MIT
- AlicenseNot gradedqualityDmaintenanceFetches web pages and converts them to markdown for LLM consumption, supporting chunked reading and raw content extraction.MIT
- AlicenseNot gradedqualityDmaintenanceMCP server for headless browser automation using Puppeteer, enabling AI to navigate, click, fill forms, take screenshots, and execute JavaScript on web pages.MIT
- AlicenseNot gradedqualityDmaintenanceMCP server providing 17 keyless, pay-per-use web-data tools with signed-provenance receipts, enabling AI agents to autonomously fetch, extract, and verify web content on Base mainnet.48 npmMIT