Xiaohongshu Comment Analysis Tool (XHS Comment Analyzer)

An automated UGC comment scraping and analysis tool for Xiaohongshu, designed for brand marketing teams. It supports searching notes by brand + category keywords, batch extracting comments (including sub-comments), three-dimensional analysis (keywords, sentiment, popularity), and exporting reports in Excel + JSON dual format. It runs as an MCP Server and can be directly invoked by AI clients such as TRAE / Claude / Cursor, and also supports standalone CLI operation.

Core Capabilities

• Automatic Note Search: Search by brand + category keyword combination, auto-scroll to load more, intelligently filter irrelevant content.

• Batch Comment Scraping: Open each note detail page one by one, extract complete information for parent and child comments, support resumable scraping.

• Three-dimensional Analysis Engine: jieba + TF-IDF keyword extraction, sentiment dictionary + rule-based sentiment classification, interaction × time-decay popularity scoring.

• Dual-format Report Export: 5-sheet Excel report + structured JSON data.

• AI Workflow Integration: MCP Server exposes 4 tools, supporting natural language-driven full process.

Related MCP server: Xiaohongshu (RedBook) MCP Server

Quick Start

Installation

pip install -e .
playwright install chromium

Run Tests

python -m pytest tests/ -v

CLI Usage

# 首次使用：检查登录状态（会打开浏览器，手动完成登录）
python run.py login

# 搜索并抓取评论（使用预设配置）
python run.py search

# 抓取单篇笔记评论
python run.py single "https://www.xiaohongshu.com/search_result/xxx?xsec_token=yyy"

# 对已有 JSON 重新分析
python run.py analyze data/output/report.json

MCP Configuration

Add the following to the MCP configuration of TRAE / Claude / Cursor:

{
    "mcpServers": {
        "xhs-comment-analyzer": {
            "command": "python",
            "args": ["-m", "src.mcp_server"],
            "cwd": "/path/to/xhs-comment-analyzer-package"
        }
    }
}

After configuration, AI clients can be invoked via natural language: "Help me search for Xiaoxiong Electric small appliance comments and generate an analysis report."

MCP Tool List

Project Structure

xhs-comment-analyzer-package/
├── src/                                # 源代码
│   ├── scraper/                        # 抓取层
│   │   ├── browser.py                  # Playwright 浏览器管理 (登录态持久化、反检测)
│   │   ├── comment_scraper.py          # 评论抓取核心 (搜索/单篇/批量/断点续抓)
│   │   └── models.py                   # 数据模型 (7 个 Pydantic 模型)
│   ├── analyzer/                       # 分析层
│   │   ├── keywords.py                 # 关键词提取 (jieba + TF-IDF)
│   │   ├── sentiment.py                # 情感分类 (词典 + 规则)
│   │   └── heat.py                     # 热度评分 (互动量 × 时效衰减)
│   ├── exporter/
│   │   └── excel_exporter.py           # 导出 Excel (5 Sheet) + JSON
│   └── mcp_server.py                   # MCP Server (4 个工具)
├── tests/                              # 单元测试 (38 个用例)
├── data/
│   ├── cookies/                        # 登录 cookie 持久化
│   ├── dictionaries/                   # 自定义词典
│   │   ├── domain_words.txt            # 领域词典 (69 个小家电术语)
│   │   ├── stopwords.txt               # 停用词表
│   │   ├── positive_words.txt          # 正面情感词
│   │   ├── negative_words.txt          # 负面情感词
│   │   ├── negation_words.txt          # 否定词
│   │   └── degree_adverbs.txt          # 程度副词 (词<TAB>权重)
│   └── output/                         # 导出文件 (Excel/JSON)
├── docs/                               # 产品文档
│   └── xhs-product-doc.html            # 完整产品文档 (PRD/架构/工作流/算法/接口)
├── run.py                              # CLI 入口 (login/search/single/analyze)
├── conftest.py                         # pytest 配置
├── pyproject.toml                      # 依赖管理
├── .gitignore
└── README.md

Output Format

Excel Report (5 Sheets)

JSON Report

Structured full data, including task configuration, summary statistics, keyword list, complete comment list, and file path, convenient for programmatic consumption.

Core Algorithms

Keyword Extraction (jieba + TF-IDF)

Each comment is treated as an independent document. After jieba segmentation, stop words and single-character words are filtered out. TF-IDF weights are calculated (sklearn-style smoothing), returning the top 10 keywords and their proportions. A built-in small appliance domain dictionary (69 terms) ensures compound words are not split.

Sentiment Analysis (Dictionary + Rules)

Based on positive/negative sentiment dictionaries + negation word flipping (within a 2-word window, support double negation) + intensifier weighting ("very" ×1.5, "especially" ×2.0, etc.). Normalized to the range [-1, 1], mapped to positive/negative/neutral labels.

Popularity Score (Interaction × Time Decay)

base_score = like_count × 2 + reply_count × 3 + sub_comment_count × 1
time_decay = 0.95 ^ days_ago
heat_score = (base_score × time_decay / max_raw_heat) × 100

Reply count has the highest weight (3) because replies indicate deep discussion; likes come second (2); sub-comments have the lowest weight (1). Decays 5% per day to ensure recent high-interaction comments rank higher.

Security Design

• No Credential Bypass: The tool does not fill in account passwords, simulate QR code scanning, or automatically handle CAPTCHAs.

• Human Intervention First: All login and CAPTCHA operations are performed manually by the user in a visible browser window.

• Visible Browser: Always uses headless=False, allowing users to view and intervene at any time.

• Risk Control Warning: Automatically stops after encountering CAPTCHAs 3 consecutive times to avoid triggering risk controls.

• Resumable Scraping: Supports resume after interruption, progress saved in progress.json.

• Cookie Persistence: Login state saved to xhs_cookies.json to avoid frequent logins.

Tech Stack

Product Documentation

Complete interactive product documentation is located at docs/xhs-product-doc.html. Open it in a browser to view. The documentation contains 9 chapters:

1. Product Overview

2. Product Requirements Document (PRD)

3. System Architecture

4. Workflow

5. Core Algorithms

6. Security and Anti-Detection

7. Data Model

8. MCP Interface

9. Usage Guide

Sample Data

The data/output/ directory contains sample output from an actual run (Xiaoxiong Electric small appliance category) for reference:

• Notes: 8 (after filtering)

• Comments: 58 (including sub-comments)

• Sentiment Distribution: Positive 22.4%, Negative 10.3%, Neutral 67.2%

• Keywords: Xiaoxiong, like, steamer

Configuration and Environment Variables

• The tool runs without any environment variables or keys by default; all login states are performed manually via a visible browser, with cookies persisted to data/cookies/.

• If integrating external services (proxies, API keys, etc.) in the future, please write the configuration to a .env file (which is ignored by .gitignore), and refer to the .env.example template. Never commit real keys.

• The following directories/files are excluded by .gitignore and will not be added to version control: data/cookies/*.json (login state), data/output/* (scraped results), data/progress.json, data/error.log, .env, etc.

Directory and File Description

License

This project is open source under the MIT License. See the LICENSE file for details (if not provided, contact the author to obtain it).