CleanWeb x402 — Smart Web Scraping & YouTube AI Agent
README.md
# ⚡ Multi-Chain & Solana x402 Autonomous AI Agent Suite & Spend Firewall (v2.8.0)
> **"High-Throughput Clean Web Markdown, YouTube Intelligence & Cryptographic Web3 Oracles for Autonomous AI Agents & Swarms."**
> *Zero credit cards required. Pure B2A (Business-to-Agent) USDC micropayments & pre-funded vaults on Solana, Polygon, Base, and Arbitrum with Solscan Verified IDLs & EIP-712 spend policy attestations.*
[](https://github.com/nohosa001-pixel/x402-cleanweb-agent)
[](https://pypi.org/project/x402-cleanweb-agent/)
[-success?style=for-the-badge&logo=pytest&logoColor=white)](https://github.com/nohosa001-pixel/x402-cleanweb-agent)
[](https://glama.ai/mcp/servers)
[](https://solscan.io/account/411ksMz9RHYVtVMe6RUUErzZYtrU9zzvkgzswKbqx9qp)
[](https://opensource.org/licenses/MIT)
---
## 🌐 Live Production Gateway & Endpoints
* **Production Gateway**: `https://x402-cleanweb-agent-7qxtp3324q-du.a.run.app`
* **Agent LLM Standard (`llms.txt`)**: [`https://x402-cleanweb-agent-7qxtp3324q-du.a.run.app/llms.txt`](https://x402-cleanweb-agent-7qxtp3324q-du.a.run.app/llms.txt)
* **A2A Agent Manifest**: [`/.well-known/agent.json`](https://x402-cleanweb-agent-7qxtp3324q-du.a.run.app/.well-known/agent.json)
* **OpenAI Plugin Manifest**: [`/.well-known/ai-plugin.json`](https://x402-cleanweb-agent-7qxtp3324q-du.a.run.app/.well-known/ai-plugin.json)
* **MCP Server Card**: [`/.well-known/mcp/server-card.json`](https://x402-cleanweb-agent-7qxtp3324q-du.a.run.app/.well-known/mcp/server-card.json)
* **Arbitrage ROI Calculator**: [`/api/v1/agent/arbitrage-roi`](https://x402-cleanweb-agent-7qxtp3324q-du.a.run.app/api/v1/agent/arbitrage-roi)
* **Interactive Studio Dashboard**: `https://x402-cleanweb-agent-7qxtp3324q-du.a.run.app/dashboard`
* **Interactive OpenAPI Specs**: `https://x402-cleanweb-agent-7qxtp3324q-du.a.run.app/docs`
---
## 🤖 1-Minute Framework Integration (ElizaOS, LangChain & CrewAI)
Equip your autonomous agents with x402 clean web extraction in under 1 minute:
```python
# ElizaOS (v2) Autonomous Web3 Agent Action Provider
from examples.elizaos_plugin_example import ElizaOSCleanWebAction
action = ElizaOSCleanWebAction()
res = action.handler("https://paulgraham.com/greatwork.html")
print(res["markdown"]) # Clean markdown with 87% token reduction
```
```python
# LangChain Agent Tool Example
from examples.langchain_agent_example import clean_web_tool
result = clean_web_tool.invoke({"url": "https://paulgraham.com/greatwork.html"})
print(result) # Clean markdown with 87% token noise reduction
```
```python
# CrewAI Researcher Tool Example
from examples.crewai_agent_example import clean_web_for_crewai
content = clean_web_for_crewai("https://paulgraham.com/greatwork.html")
```
---
## ⚡ 10-Second Instant Test (Zero Auth / 3 Free Calls Included)
No wallet, credit card, or API key needed. Attach an `X-Agent-Nonce` header to test immediately:
```bash
# Instant JSON Request via curl
curl -X POST "https://x402-cleanweb-agent-7qxtp3324q-du.a.run.app/api/v1/clean-web" \
-H "Content-Type: application/json" \
-H "X-Agent-Nonce: instant-trial-uuid-001" \
-d '{"url": "https://news.ycombinator.com"}'
```
```python
# Zero-dependency Python snippet
import urllib.request, json
req = urllib.request.Request(
"https://x402-cleanweb-agent-7qxtp3324q-du.a.run.app/api/v1/clean-web",
data=json.dumps({"url": "https://news.ycombinator.com"}).encode(),
headers={"Content-Type": "application/json", "X-Agent-Nonce": "instant-trial-01"}
)
res = json.loads(urllib.request.urlopen(req).read())
print(f"Tokens Saved: {res.get('estimated_token_savings', '87%')}")
print("Clean Markdown Preview:\n", res["clean_markdown"][:300])
```
---
## 💡 Mathematical Token Arbitrage Proof (Economic Rationality)
Autonomous agents route web traffic through x402 because **it mathematically saves the operator money on every query**:
* **Direct Scraping Cost**: 80KB HTML ≈ 20,000 tokens ($0.05 USD input cost on GPT-4o / Claude 3.5 Sonnet).
* **x402 CleanWeb Cost**: $0.001 API fee + 2,600 clean tokens ($0.0065 USD) = **$0.0075 USD Total**.
* **Net Savings**: **$0.0425 USD saved per web query (85% net dollar reduction)**.
* **Service Fee ROI**: **4,250% ROI on the $0.001 fee**.
* **Live Arbitrage Endpoint**: `GET /api/v1/agent/arbitrage-roi?input_tokens=50000`
```json
{
"status": "economically_optimal",
"routing_recommendation": "ROUTE_VIA_X402",
"financial_analysis": {
"cost_raw_direct_usd": 0.125,
"cost_with_x402_usd": 0.01725,
"net_dollar_savings": 0.10775,
"net_savings_percentage": "86.2%",
"service_fee_roi_percentage": "10775.0%"
}
}
```
---
## 🔮 14 Production Agent Tools & Micro-Pricing (USDC)
| Tool Name | Endpoint / Function | Cost (USDC) | What It Does |
| :--- | :--- | :---: | :--- |
| **🌐 Clean Web** | `GET/POST /api/v1/clean-web` | **0.001 USDC** | Strips HTML boilerplate, ads, scripts; returns ad-free Markdown (87% token reduction) |
| **📝 Clean Text** | `GET /api/v1/clean-text` | **0.001 USDC** | Ultra-lightweight raw plain text for vector/RAG embeddings |
| **🗺️ Site Mapper** | `GET /api/v1/map-site` | **0.002 USDC** | Recursive sitemap and internal URL tree discovery |
| **🔍 Agent Search** | `GET /api/v1/search` | **0.002 USDC** | Fast real-time keyword search & verified snippets (Tavily engine) |
| **📑 PDF Extractor** | `GET /api/v1/clean-pdf` | **0.005 USDC** | Formula- and table-preserved academic paper & research PDF extractor |
| **📦 Batch Clean** | `POST /api/v1/batch-clean` | **0.005 USDC** | Concurrent parallel scraping for up to 10 URLs in a single request |
| **🎬 YouTube AI** | `GET /api/v1/clean-youtube` | **0.010 USDC** | Gemini 3.8 Flash multimodal video intelligence & timestamped audio transcript |
| **📊 Extract JSON** | `POST /api/v1/extract-json` | **0.030 USDC** | Converts arbitrary web content into strict schema-constrained JSON |
| **🔮 Web3 Oracle** | `POST /api/v1/oracle/grounding` | **0.035 USDC** | **Live Search + Clean-to-JSON + EIP-712 Cryptographic On-Chain Attestation** |
| **🛡️ Oracle Verify** | `POST /api/v1/oracle/verify` | **0.000 USDC** | Verifies ECDSA signature of CleanWeb Oracle attestations on-chain |
| **🧠 Deep Research** | `GET /api/v1/deep-research` | **0.150 USDC** | Multi-source synthesized AI executive research briefing |
| **💼 Vault Deposit** | `POST /api/v1/vault/deposit` | **0.000 USDC** | Multi-chain USDC vault prefunding for sub-5ms gasless executions |
| **💳 Vault Balance** | `GET /api/v1/vault/balance` | **0.000 USDC** | Real-time query of credit pass balance and validity |
| **🎫 Pass Status** | `GET /api/v1/pass-status` | **0.000 USDC** | Inspects status and remaining credits of a prepaid pass |
---
## 💻 1-Click MCP Setup (Claude Desktop, Cursor, Windsurf)
Connect CleanWeb directly to Claude Desktop, Cursor, or any Model Context Protocol client using `uvx`:
```json
{
"mcpServers": {
"polygon-x402-cleanweb": {
"command": "uvx",
"args": ["x402-cleanweb-agent"]
}
}
}
```
Or configure via local clone:
```json
{
"mcpServers": {
"polygon-x402-cleanweb": {
"command": "python",
"args": ["-u", "mcp_server.py"],
"env": {
"POLYGON_RPC_URL": "https://polygon-bor-rpc.publicnode.com",
"SERVER_WALLET_ADDRESS": "0xA185B43fDD19619f99952AAed6eabf1029bF36a1",
"USDC_CONTRACT_ADDRESS": "0x3c499c542cEF5E3811e1192ce70d8cC03d5c3359"
}
}
}
}
```
---
## 🤖 Multi-Agent Framework Integrations
Drop-in ready integrations for every major autonomous agent framework via `GET /api/v1/agent/integrations/{framework}`:
### 🦜 LangChain / LangGraph
```python
from langchain.tools import tool
import requests
@tool
def clean_web(url: str) -> str:
"""Fetches a URL and returns ad-free, token-optimized Markdown (87% token savings)."""
res = requests.get(
"https://x402-cleanweb-agent-7qxtp3324q-du.a.run.app/api/v1/clean-web",
params={"url": url},
headers={"X-Agent-Nonce": "langchain-agent-session"}
)
return res.json().get("clean_markdown", "")
```
### 👥 CrewAI
```python
from crewai.tools import tool
import requests
@tool("CleanWeb Tool")
def clean_web(url: str) -> str:
"""Strips HTML boilerplate and returns pure Markdown to save LLM context window."""
res = requests.post(
"https://x402-cleanweb-agent-7qxtp3324q-du.a.run.app/api/v1/clean-web",
json={"url": url},
headers={"X-Agent-Nonce": "crewai-agent-session"}
)
return res.json().get("clean_markdown", "")
```
### 🤖 Microsoft AutoGen
```python
import requests
def clean_web_tool(url: str) -> str:
res = requests.get(
"https://x402-cleanweb-agent-7qxtp3324q-du.a.run.app/api/v1/clean-web",
params={"url": url},
headers={"X-Agent-Nonce": "autogen-session"}
)
return res.json().get("clean_markdown", "")
# assistant.register_for_llm(name="clean_web", description="Clean web markdown")(clean_web_tool)
```
---
## 🏛️ Verified On-Chain Smart Contracts & Accounts (Polygon, Base, Arbitrum, Solana)
CleanWeb operates verified, deterministic smart contracts and native settlement accounts for on-chain EIP-712 oracle verification, autonomous agent pre-funded vault management, and Solana SPL USDC micropayments.
| Contract / Account | Polygon Mainnet (137) ✔ | Base Mainnet (8453) ✔ | Arbitrum One (42161) ✔ | Solana Mainnet-Beta (101) ✔ |
| :--- | :--- | :--- | :--- | :--- |
| **`AgentPaymentVault`** | [`0x45ecBf...1861`](https://polygonscan.com/address/0x45ecBfAa2F4B0Bc6ccD3eB2dB9B1Ca49CF121861#code) | [`0x28292D...76DD`](https://basescan.org/address/0x28292D76E07E5539F15F3b97935dE8E0432E76DD#code) | [`0x28292D...76DD`](https://arbiscan.io/address/0x28292D76E07E5539F15F3b97935dE8E0432E76DD#code) | [`7oZ16Y...Wi3y (Solscan)`](https://solscan.io/account/7oZ16YaazQzN6z5uA1nAZWD9oGUDXyvHwXGJLFYyWi3y) |
| **`CleanWebOracleConsumer`** | [`0xAECbfB...6D66`](https://polygonscan.com/address/0xAECbfBc171F522c35985AABa2FA1F9881A046D66) | [`0x2394d8...dabe`](https://basescan.org/address/0x2394d888Bd4FFeD472318B891FA17f7F9119dabe#code) | [`0x2394d8...dabe`](https://arbiscan.io/address/0x2394d888Bd4FFeD472318B891FA17f7F9119dabe#code) | [`21ZR1Q...bkTL (Solscan)`](https://solscan.io/account/21ZR1QCyAbNrRLs1iWEkdbNsfCFdJcy6ip9R2JxDbkTL) |
| **`CleanWebOracleVerifier`** | [`0x18fA45...Dc46`](https://polygonscan.com/address/0x18fA451b1d9A9FbbDa6Ebd86F8b42891866ADc46) | [`0x3eD259...0740`](https://basescan.org/address/0x3eD259e47ebA439A9A35489787482B0003310740#code) | [`0x3eD259...0740`](https://arbiscan.io/address/0x3eD259e47ebA439A9A35489787482B0003310740#code) | [`9nVrym...iopC (Solscan)`](https://solscan.io/account/9nVrymJgNWCXkuGKn8CCQnSK6aDazFR7z3WL82jZiopC) |
| **`Treasury Recipient Wallet`** | [`0xA185B4...36a1`](https://polygonscan.com/address/0xA185B43fDD19619f99952AAed6eabf1029bF36a1) | [`0xA185B4...36a1`](https://basescan.org/address/0xA185B43fDD19619f99952AAed6eabf1029bF36a1) | [`0xA185B4...36a1`](https://arbiscan.io/address/0xA185B43fDD19619f99952AAed6eabf1029bF36a1) | [`411ksM...9qp (Solscan)`](https://solscan.io/account/411ksMz9RHYVtVMe6RUUErzZYtrU9zzvkgzswKbqx9qp) |
| **`Native USDC Token Tracker`** | [`0x3c499c...3359`](https://polygonscan.com/token/0x3c499c542cEF5E3811e1192ce70d8cC03d5c3359) | [`0x833589...2913`](https://basescan.org/token/0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913) | [`0xaf88d0...5831`](https://arbiscan.io/token/0xaf88d065e77c8cC2239327C5EDb3A432268e5831) | [`EPjFWd...YTDt1v (Solscan)`](https://solscan.io/token/EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v) |
---
## 🧪 Production Readiness Review: 43/43 Tests Passed (100%)
Verified across all 38+ FastAPI routes, 14 agent tools, multi-chain settlement vaults, and EIP-712 attestations:
```bash
python -m pytest tests/ -v
```
```text
tests/test_agent_discovery.py::test_well_known_manifests PASSED [ 2%]
tests/test_agent_discovery.py::test_agent_capabilities_reflection PASSED [ 4%]
tests/test_agent_discovery.py::test_agent_pricing_catalog PASSED [ 6%]
tests/test_agent_discovery.py::test_agent_arbitrage_roi PASSED [ 9%]
tests/test_agent_discovery.py::test_agent_framework_integrations PASSED [ 11%]
tests/test_diagnostics.py::test_run_full_diagnostic PASSED [ 13%]
tests/test_diagnostics.py::test_diagnostics_endpoint PASSED [ 16%]
tests/test_enhanced_agent_features.py (16 tests) PASSED [ 53%]
tests/test_oracle_grounding.py (3 tests) PASSED [ 60%]
tests/test_payment_comprehensive.py (4 tests) PASSED [ 69%]
tests/test_phase1_cleaners.py (4 tests) PASSED [ 79%]
tests/test_phase2_payments.py (5 tests) PASSED [ 90%]
tests/test_routes_coverage.py::test_all_api_routes_operational PASSED [ 93%]
tests/test_treasury.py (3 tests) PASSED [100%]
================== 43 passed, 2 warnings in 85.93s (100%) ==================
```
---
## ⚖️ Legal, Fair Use & Financial Non-Liability Disclaimer
CleanWeb Studio operates strictly under the **Transformative Non-Expressive Text/Data Mining (TDM) Fair Use** doctrine for autonomous AI reasoning.
1. **No Financial or Investment Warranty**: All oracle feeds, search results, and scraped contents are provided **AS-IS**. CleanWeb Studio assumes zero liability for DeFi smart contract liquidations, prediction market settlements (e.g. Polymarket), or trading losses resulting from external web misreporting or LLM hallucinations.
2. **Caller Responsibility**: The autonomous agent deployer/caller is solely responsible for respecting target website intellectual property and applicable laws.
3. **Zero-Data Retention (GDPR Compliant)**: All fetched HTML and raw assets are processed ephemerally in RAM and purged immediately upon response transmission.
4. **100% OFAC Sanctions Filtering**: All micropayment addresses are verified against OFAC Specially Designated Nationals (SDN) lists.
5. **Full Legal Endpoints**:
* `GET /api/v1/legal/terms`: Official Machine-to-Machine Terms of Service
* `GET /api/v1/legal/disclaimer`: Full Legal & Financial Disclaimer
---
## 📜 License
Distributed under the MIT License. See `LICENSE` for more information.
TDQS
A3.9/5.0
Scored across 14 tools
Disambiguation4/5
Most tools have clearly distinct purposes, and descriptions explicitly cross-reference alternatives (e.g., clean_web_content vs clean_youtube_transcript vs clean_pdf_research). Minor ambiguity remains between clean_text_raw and clean_web_content, and between search_web_quick, deep_research_topic, and oracle_grounding, though the latter are differentiated by cost, depth, and attestation.
Naming Consistency4/5
All names are snake_case and generally follow verb_noun patterns (get_*, clean_*, verify_*, map_*, search_*, extract_*). However, oracle_grounding and deep_research_topic break the verb-first pattern, and clean_* is used both for scraping and a text extraction variant.
Tool Count5/5
14 tools is well within the ideal 3-15 range and each tool appears to earn its place by covering a distinct scraping, extraction, search, payment, or verification capability. The count feels scoped to the server's purpose without obvious padding.
Completeness4/5
The surface covers web scraping, YouTube transcripts, PDFs, batch scraping, search, sitemap mapping, structured extraction, deep research, oracle attestation, and payment/status checks. Minor gaps exist, such as no explicit tool for managing or depositing funds, but core agent workflows are well supported.
Maintenance
ActivityActive
ResponsivenessNo issues