Skip to main content
Glama

Chameleon 🦎

Python License: MIT CI Tests Ruff Mypy GitHub stars GitHub forks

面向真实世界反爬、动态渲染、异常容错的 AI Agent 爬虫工具。提供 MCP Server + REST API + CLI 三套 Agent 友好接口,核心理念:分层对抗、自适应降级、Agent 友好输出。

核心能力

  • 6 级反爬自动升级:裸请求 → UA/Header 伪装 → 代理轮换 → TLS 指纹模拟 → 浏览器渲染 → 行为模拟/验证码,逐级升级,全程可观测

  • 三引擎架构:HTTP 引擎(httpx/curl_cffi)、浏览器引擎(Playwright + stealth + 行为模拟)、API 逆向引擎

  • 处理管线:HTML 清洗 → Markdown 转换(token 预算裁剪)→ 结构化提取(CSS/XPath/LLM/Hybrid/表格)→ 动态 Pydantic 校验

  • 深度爬取:BFS/DFS/Adaptive 策略、每域名限速、ETag 增量采集、断点续爬、robots.txt 合规

  • Agent 原生:MCP 12 工具自动发现、统一结构化输出(status/content/metadata)、SSRF 防护 + 审计日志

Related MCP server: Selenium MCP Server

接口形态

形态

说明

MCP Server

stdio 传输,12 个工具,SSRF 防护 + 输出截断

REST API

FastAPI,与 MCP 工具一一对应,API Key 认证 + WebSocket 进度

CLI

chameleon scrape/crawl/extract/map/search ...,退出码语义化

Python SDK

Chameleon 服务门面,scrape/crawl/extract/batch/search 全覆盖

快速开始

uv sync
uv run chameleon scrape https://example.com -o markdown          # CLI
uv run python -m chameleon.interfaces.mcp_server                 # MCP (stdio)
uv run uvicorn chameleon.interfaces.rest_api:app --port 8000     # REST API

MCP 配置(Claude Code / opencode)

{
  "mcpServers": {
    "chameleon": {
      "command": "uv",
      "args": ["run", "python", "-m", "chameleon.interfaces.mcp_server"]
    }
  }
}

Agent 自动发现工具后可直接说:"抓取 https://example.com/products 的所有产品名称和价格,输出 JSON"。

架构一览

Agent → MCP/REST/CLI → Chameleon 门面
  ├─ SmartRouter(L0-L6 升级链 + 策略记忆)
  ├─ Pipeline(清洗 → Markdown → 提取 → 校验)
  └─ DeepCrawler + Scheduler(策略遍历/限速/断点)

详见 docs/architecture.md。

测试

uv run pytest          # 229 个测试(含 Playwright 真实渲染、反爬模拟站对抗)
uv run ruff check src tests
uv run mypy src

边界与声明

反爬对抗是持续的军备竞赛,本工具不保证对任何站点必然成功:

  • 持续演进的验证码(滑块/设备指纹/风控评分)可能击败任何自动化手段

  • 内置升级链全程透明可观测——结果中记录实际使用的引擎与升级层级,Agent 可据此判断可信度

  • 住宅代理与打码服务为第三方付费依赖,请按预算使用

  • 仅在获得授权的范围内使用:遵守 robots.txt、站点服务条款、当地法律(GDPR/个保法)与限速要求。滥用造成的后果由使用者自行承担

文档

开源

MIT


⭐ 求个 Star

如果这个项目对你有帮助,欢迎点右上角给个 Star ⭐ —— 这是对开源作者最大的鼓励,也能让更多人发现它!

If you find Chameleon useful, please consider giving it a star ⭐ — it means a lot!

Star History

Star History Chart

Available Tools

12 tools
batch_scrapeB

批量采集多个 URL。当用户给出多个网址时使用。

Inputs: urls (required), concurrency, output_format. Returns: object

check_proxyC

检测代理 IP 的可用性和出口信息。

Inputs: proxy_url (required). Returns: object

crawl_siteB

深度爬取整个网站,返回 job_id。当用户要求采集整个网站或栏目时使用。

Inputs: url (required), strategy, max_depth, max_pages. Returns: object

diagnose_siteB

诊断网站反爬能力和推荐策略。首次接触新站点时使用。

Inputs: url (required). Returns: object

extract_dataA

基于 JSON Schema 从网页提取结构化数据。当用户要求特定字段(价格/标题/日期等)时使用。

Inputs: url (required), schema (required), strategy. Returns: object

get_job_statusB

查询异步爬取任务状态。

Inputs: job_id (required). Returns: object

get_network_logA

获取页面网络请求日志(用于 API 发现)。filter_type: xhr|fetch|all

Inputs: url (required), filter_type. Returns: object

get_robots_txtB

获取网站的 robots.txt 规则。

Inputs: url (required). Returns: object

get_screenshotA

获取网页截图(base64 PNG)。当用户想看页面外观/布局时使用。

Inputs: url (required), full_page. Returns: object

map_siteB

快速发现网站所有 URL(不抓取内容)。

Inputs: url (required), sitemap_only. Returns: object

scrape_urlA

采集单个 URL 并转为 Markdown/JSON。当用户给出具体网址或询问某页面内容时使用。 不要用于:搜索互联网(用 search_web)。

Inputs: url (required), mode, schema, wait_for, +3 more. Returns: object

search_webB

搜索互联网并返回结果页面的 Markdown。当用户询问实时信息/最新内容时使用。

Inputs: query (required), language, max_results. Returns: object

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.6/5.0

    Scored across 12 tools

    Disambiguation5/5

    Each tool targets a distinct resource and action: single URL scraping, batch scraping, site crawling, URL mapping, structured extraction, web search, screenshots, proxy checks, job status, robots.txt, and network logs. Descriptions clearly distinguish between overlapping concepts like crawl_site vs map_site and scrape_url vs extract_data.

    Naming Consistency5/5

    All tool names follow a consistent verb_noun pattern using snake_case (e.g., scrape_url, diagnose_site, get_screenshot). The only slight deviation is batch_scrape, but it still maintains the object-oriented naming style and is clearly readable.

    Tool Count5/5

    12 tools is well within the ideal 3-15 range and each tool serves a distinct purpose in the web scraping domain. The count feels comprehensive without being bloated.

    Completeness4/5

    The tool set covers the core scraping workflow: discover (map_site), analyze (diagnose_site, get_robots_txt), fetch (scrape_url, crawl_site, batch_scrape), extract (extract_data), and verify (get_screenshot, get_network_log). Minor gaps include no explicit job cancellation/management tool and no export/storage utility, but agents can work around these.

    Maintenance

    ActivitySlowing
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      C
      maintenance
      A local MCP server that lets AI agents bypass bot detection, geo-restrictions, and JavaScript rendering challenges when scraping the web, backed by ScraperAPI's services
      28
      5
      MIT
    • F
      license
      Not graded
      quality
      D
      maintenance
      MCP server for crawling and extracting data from web pages using Selenium with CSS/XPath selectors. Supports 17 tools including smart extraction, pagination, infinite scroll, and screenshots.
      3
      -
    • A
      license
      Not graded
      quality
      D
      maintenance
      An MCP server that gives AI tools a reliable way to fetch content from the web, handling anti-bot protection and JavaScript-rendered pages.
      1,769 npm
      MIT
    • A
      license
      Not graded
      quality
      D
      maintenance
      MCP server for web scraping and browser automation, enabling AI agents to extract clean, token-efficient content from web pages.
      1
      MIT