chameleon
Chameleon is an agent-friendly web scraping MCP server exposing 12 tools for fetching, crawling, extracting, searching, and diagnosing sites against anti-bot defenses.
scrape_url — fetch a single URL and convert it to Markdown/JSON, with optional CSS/LLM extraction prompt, wait condition, proxy region, and schema validation
crawl_site — deep-crawl an entire site (BFS/DFS/adaptive, depth and page limits), returns a job_id
map_site — quickly discover all URLs on a site without fetching content (optionally sitemap-only)
extract_data — pull structured data from a page using a JSON Schema and a chosen strategy
search_web — search the internet and return result pages as Markdown (language, max results)
batch_scrape — scrape many URLs at once with configurable concurrency
get_screenshot — capture a page as a base64 PNG (viewport or full page)
diagnose_site — probe a site's anti-bot defenses and recommend a crawling strategy
check_proxy — test a proxy IP's availability and exit information
get_job_status — poll the status of an async crawl job
get_robots_txt — retrieve a site's robots.txt rules for compliance
get_network_log — inspect a page's XHR/fetch network requests to discover underlying APIs
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@chameleon Scrape https://example.com/products and extract product names and prices as JSON"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Chameleon 🦎
面向真实世界反爬、动态渲染、异常容错的 AI Agent 爬虫工具。提供 MCP Server + REST API + CLI 三套 Agent 友好接口,核心理念:分层对抗、自适应降级、Agent 友好输出。
核心能力
6 级反爬自动升级:裸请求 → UA/Header 伪装 → 代理轮换 → TLS 指纹模拟 → 浏览器渲染 → 行为模拟/验证码,逐级升级,全程可观测
三引擎架构:HTTP 引擎(httpx/curl_cffi)、浏览器引擎(Playwright + stealth + 行为模拟)、API 逆向引擎
处理管线:HTML 清洗 → Markdown 转换(token 预算裁剪)→ 结构化提取(CSS/XPath/LLM/Hybrid/表格)→ 动态 Pydantic 校验
深度爬取:BFS/DFS/Adaptive 策略、每域名限速、ETag 增量采集、断点续爬、robots.txt 合规
Agent 原生:MCP 12 工具自动发现、统一结构化输出(status/content/metadata)、SSRF 防护 + 审计日志
Related MCP server: Selenium MCP Server
接口形态
形态 | 说明 |
MCP Server | stdio 传输,12 个工具,SSRF 防护 + 输出截断 |
REST API | FastAPI,与 MCP 工具一一对应,API Key 认证 + WebSocket 进度 |
CLI |
|
Python SDK |
|
快速开始
uv sync
uv run chameleon scrape https://example.com -o markdown # CLI
uv run python -m chameleon.interfaces.mcp_server # MCP (stdio)
uv run uvicorn chameleon.interfaces.rest_api:app --port 8000 # REST APIMCP 配置(Claude Code / opencode)
{
"mcpServers": {
"chameleon": {
"command": "uv",
"args": ["run", "python", "-m", "chameleon.interfaces.mcp_server"]
}
}
}Agent 自动发现工具后可直接说:"抓取 https://example.com/products 的所有产品名称和价格,输出 JSON"。
架构一览
Agent → MCP/REST/CLI → Chameleon 门面
├─ SmartRouter(L0-L6 升级链 + 策略记忆)
├─ Pipeline(清洗 → Markdown → 提取 → 校验)
└─ DeepCrawler + Scheduler(策略遍历/限速/断点)测试
uv run pytest # 229 个测试(含 Playwright 真实渲染、反爬模拟站对抗)
uv run ruff check src tests
uv run mypy src边界与声明
反爬对抗是持续的军备竞赛,本工具不保证对任何站点必然成功:
持续演进的验证码(滑块/设备指纹/风控评分)可能击败任何自动化手段
内置升级链全程透明可观测——结果中记录实际使用的引擎与升级层级,Agent 可据此判断可信度
住宅代理与打码服务为第三方付费依赖,请按预算使用
仅在获得授权的范围内使用:遵守 robots.txt、站点服务条款、当地法律(GDPR/个保法)与限速要求。滥用造成的后果由使用者自行承担
文档
开源
⭐ 求个 Star
如果这个项目对你有帮助,欢迎点右上角给个 Star ⭐ —— 这是对开源作者最大的鼓励,也能让更多人发现它!
If you find Chameleon useful, please consider giving it a star ⭐ — it means a lot!
Star History
Available Tools
12 toolsbatch_scrapeB
批量采集多个 URL。当用户给出多个网址时使用。
Inputs: urls (required), concurrency, output_format. Returns: object
check_proxyC
检测代理 IP 的可用性和出口信息。
Inputs: proxy_url (required). Returns: object
crawl_siteB
深度爬取整个网站,返回 job_id。当用户要求采集整个网站或栏目时使用。
Inputs: url (required), strategy, max_depth, max_pages. Returns: object
diagnose_siteB
诊断网站反爬能力和推荐策略。首次接触新站点时使用。
Inputs: url (required). Returns: object
extract_dataA
基于 JSON Schema 从网页提取结构化数据。当用户要求特定字段(价格/标题/日期等)时使用。
Inputs: url (required), schema (required), strategy. Returns: object
get_job_statusB
查询异步爬取任务状态。
Inputs: job_id (required). Returns: object
get_network_logA
获取页面网络请求日志(用于 API 发现)。filter_type: xhr|fetch|all
Inputs: url (required), filter_type. Returns: object
get_robots_txtB
获取网站的 robots.txt 规则。
Inputs: url (required). Returns: object
get_screenshotA
获取网页截图(base64 PNG)。当用户想看页面外观/布局时使用。
Inputs: url (required), full_page. Returns: object
map_siteB
快速发现网站所有 URL(不抓取内容)。
Inputs: url (required), sitemap_only. Returns: object
scrape_urlA
采集单个 URL 并转为 Markdown/JSON。当用户给出具体网址或询问某页面内容时使用。 不要用于:搜索互联网(用 search_web)。
Inputs: url (required), mode, schema, wait_for, +3 more. Returns: object
search_webB
搜索互联网并返回结果页面的 Markdown。当用户询问实时信息/最新内容时使用。
Inputs: query (required), language, max_results. Returns: object
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 12 tools
Each tool targets a distinct resource and action: single URL scraping, batch scraping, site crawling, URL mapping, structured extraction, web search, screenshots, proxy checks, job status, robots.txt, and network logs. Descriptions clearly distinguish between overlapping concepts like crawl_site vs map_site and scrape_url vs extract_data.
All tool names follow a consistent verb_noun pattern using snake_case (e.g., scrape_url, diagnose_site, get_screenshot). The only slight deviation is batch_scrape, but it still maintains the object-oriented naming style and is clearly readable.
12 tools is well within the ideal 3-15 range and each tool serves a distinct purpose in the web scraping domain. The count feels comprehensive without being bloated.
The tool set covers the core scraping workflow: discover (map_site), analyze (diagnose_site, get_robots_txt), fetch (scrape_url, crawl_site, batch_scrape), extract (extract_data), and verify (get_screenshot, get_network_log). Minor gaps include no explicit job cancellation/management tool and no export/storage utility, but agents can work around these.
Maintenance
Related MCP Connectors
Web scraping MCP server — scrape, extract structured data, screenshot any site with anti-bot bypass.
One MCP server for 180+ live web-data APIs returning clean JSON from sites that block scrapers.
Web scraping, extraction and search for AI agents. Hosted MCP with workspace API keys.
One MCP server for all web data. Scrape, search, crawl, and AI research in one install.
Related MCP Servers
AlicenseAqualityCmaintenanceA local MCP server that lets AI agents bypass bot detection, geo-restrictions, and JavaScript rendering challenges when scraping the web, backed by ScraperAPI's services285MIT- FlicenseNot gradedqualityDmaintenanceMCP server for crawling and extracting data from web pages using Selenium with CSS/XPath selectors. Supports 17 tools including smart extraction, pagination, infinite scroll, and screenshots.3-
- AlicenseNot gradedqualityDmaintenanceAn MCP server that gives AI tools a reliable way to fetch content from the web, handling anti-bot protection and JavaScript-rendered pages.1,769 npmMIT
- AlicenseNot gradedqualityDmaintenanceMCP server for web scraping and browser automation, enabling AI agents to extract clean, token-efficient content from web pages.1MIT