the-data-collector
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@the-data-collectorsearch hacker news for AI trends"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
The Data Collector
Web scraping APIs for Bluesky, Substack, and Hacker News with x402 micropayment support. Built with FastAPI.
Live: https://frog03-20494.wykr.es
Features
Search Bluesky posts (AT Protocol), Substack newsletters, and Hacker News stories
Returns structured JSON with engagement metrics
x402 micropayments ($0.05 USDC on Base per call) — no account needed
API key authentication for regular use
A2A Agent Card and MCP discovery endpoints
OpenAPI spec with x402 payment metadata
Related MCP server: x402search
API Endpoints
Method | Endpoint | Description | Price |
POST |
| Search Bluesky posts by keyword | $0.05 |
POST |
| Scrape Substack newsletter articles | $0.05 |
POST |
| Search Hacker News stories | $0.05 |
Quick Start
# Clone
git clone https://github.com/MarcinDudekDev/the-data-collector.git
cd the-data-collector
# Install
python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt
# Configure
cp .env.example .env
# Edit .env with your APIFY_TOKEN and API_KEY
# Run
uvicorn server:app --host 0.0.0.0 --port 8001Docker
docker build -t the-data-collector .
docker run -p 8001:8001 --env-file .env the-data-collectorEnvironment Variables
Variable | Required | Description |
| Yes | Apify API token for running scrapers |
| No | API key for authenticated access (X-API-Key header) |
| No | Public URL of the server (default: https://frog03-20494.wykr.es) |
| No | Wallet address for x402 payments |
| No | Price per call in USDC atomic units (default: 50000 = $0.05) |
Authentication
x402 Micropayments (no account needed)
Send a POST request without credentials. You'll receive a 402 response with payment requirements. Pay $0.05 USDC on Base — settlement is instant.
# First call returns 402 with payment details
curl -X POST https://frog03-20494.wykr.es/api/hn/search \
-H "Content-Type: application/json" \
-d '{"searchTerms": ["AI agents"]}'API Key
curl -X POST https://frog03-20494.wykr.es/api/hn/search \
-H "Content-Type: application/json" \
-H "X-API-Key: your-key" \
-d '{"searchTerms": ["AI agents"], "maxResults": 5}'Discovery Endpoints
Endpoint | Protocol |
| MCP (Model Context Protocol) |
| A2A (Agent-to-Agent) |
| x402 payment discovery |
| OpenAPI 3.1 spec |
| Health check |
MCP Client Configuration
{
"mcpServers": {
"the-data-collector": {
"url": "https://frog03-20494.wykr.es/.well-known/mcp.json"
}
}
}License
MIT
This server cannot be deployed
Maintenance
Related MCP Connectors
Search ATProto writing, annotations, identity, agents, and forum posts. 12 read-only tools.
1 Google Search endpoints. Pay per call in USDC via x402.
X/Twitter reads, search, monitors and posting. Pay-per-call in USDC — no signup, no API keys.
Search settlement-verified x402 APIs. Discovery is free. Agents pay the provider in USDC on Base.
Related MCP Servers
- AlicenseAqualityDmaintenanceEnables browsing Hacker News, searching discussions, analyzing users, and tracking tech trends with zero setup required—no API keys or authentication needed.523 npm6MIT
- AlicenseAqualityDmaintenanceSearch 14,000+ x402-enabled HTTP APIs by keyword. Agents pay $0.01 USDC per search via x402 micropayments on Base mainnet — no API keys or accounts required.139 npm2MIT
- AlicenseAqualityCmaintenancePaid web research MCP tools for autonomous agents: search, page extraction, citations, and diff monitoring through a live x402 API. Unpaid calls return the Base USDC payment requirement so agents can pay and retry safely.41MIT
- AlicenseNot gradedqualityDmaintenancePay-per-call web search for AI agents, settled in USDC on Base via the x402 protocol. No API key or subscription required; users fund a wallet and get a web_search tool.7 npm1MIT