JD-PULL · JD Crawling · Analysis · MCP Tool

An open-source job JD crawling + rule-based analysis + job portrait + MCP server tool. Anyone can clone it locally, crawl and analyze their own data, and expose the data to any MCP-capable external AI such as Claude / DeepSeek agent / Pi agent / Cursor via MCP (Model Context Protocol).

┌──────────────────────────┐      ┌──────────────────────────┐
│  外部 AI（任何 MCP 客户端） │ ───► │  本机 jd-mcp 服务器        │
│  Claude / DeepSeek / Pi  │  MCP  │  ┌────────────────────┐  │
│  / Cursor …              │       │  │ 只读查询 · 写操作     │  │
└──────────────────────────┘       │  └────────┬───────────┘  │
                                    └──────────┼───────────────┘
                                               │ 读写
                                    ┌──────────▼───────────────┐
                                    │  你的数据目录（JD_DATA_DIR）│
                                    │  本地文件夹 或 私有 git 仓库 │
                                    └──────────────────────────┘

Crawled results exist only in your local data/ directory or a data repository you specify.

Features

• JD crawling: 51job (Playwright real-browser driven, default system Edge) low-frequency polite crawling; Boss Zhipin / Lagou / Liepin reserved interfaces (require login/signature, not implemented in this release);

• Rule-based analysis: classifies JDs into job categories using a classification dictionary, extracts high-frequency skills and responsibility terms;

• LLM refinement (optional, multi-vendor): six vendors — Claude / OpenAI / Gemini / DeepSeek / Qwen / Ollama — automatically adapted to each vendor's structured output via a "capability ladder";

• Job portrait: generates skill / salary / experience / education distribution and high-frequency responsibility reports per category (JSON + Markdown);

• MCP server: dual transport over stdio and streamable-http; read-only query tools always available, write tools enabled by default (disabled with --no-write-tools), optional Bearer token auth over HTTP;

• Fully local data: JD_DATA_DIR can point to any directory or git repository; when the data directory is not a standalone git repository, sync automatically skips it to protect data from accidental commits.

Related MCP server: MCP Job Search Server

Tech Stack

Installation

Requires Python ≥ 3.10.

git clone https://github.com/NaoYUN77/JD-PULL.git
cd JD-PULL
python -m venv .venv && .venv\Scripts\activate      # Windows；macOS/Linux 用 source .venv/bin/activate
pip install -e .[dev]
# 51job 爬取走真实浏览器（默认系统 Edge，无需额外下载 chromium）
# 若想用捆绑 chromium：删除 config/settings.yaml 中 crawl.browser.channel 一行后
#   playwright install chromium

Quick Start (Get Data First)

Job sites generally have anti-crawling protections. Please crawl at low frequency, with a real browser, and only as needed; stop if blocked, and do not attempt to bypass.

# 1) 生成样本 JD（验证全链路；51job 被 WAF 拦截时用）
jdcollector seed

# 2) 真实爬取（按 config/sites.yaml 的关键词/城市；51job 需本机有 Edge 且有头窗口）
jdcollector crawl

# 3) 规则分析（可选加 LLM 精分，见下文「LLM 多供应商精分」）
jdcollector analyze

# 4) 岗位群像报告
jdcollector portrait

# 5) 静态 HTML 报告站（输出到数据目录 site/）
jdcollector view

Data is written by default to the repository root data/ (ignored by .gitignore, so it won't be committed).

Use as an MCP Server

Once started, external AIs can call your data via MCP.

# stdio（默认，给本地桌面客户端用）
jd-mcp

# 只读模式（只暴露查询，不暴露爬取/分析等写工具）
jd-mcp --no-write-tools

# HTTP（streamable-http，给远程客户端 / 其它进程用，可带 Bearer token）
jd-mcp --transport http --host 0.0.0.0 --port 8000 --token 你的token

Claude Desktop

Edit claude_desktop_config.json:

{
  "mcpServers": {
    "jdcollector": {
      "command": "jd-mcp",
      "args": ["--transport", "stdio"]
    }
  }
}

Claude Code

claude mcp add jdcollector -- jd-mcp --transport stdio
claude mcp list   # 验证已连接

Cursor

Settings → MCP → + Add global MCP server, and fill in the command:

jd-mcp --transport stdio

MCP Tools Overview

Also registers read-only resources: jd://stats, jd://portrait/{category}.

HTTP Deployment and Authentication

jd-mcp --transport http --host 0.0.0.0 --port 8000 --token 你的token
# 环境变量亦可：JD_MCP_TRANSPORT / JD_MCP_HOST / JD_MCP_PORT / JD_MCP_TOKEN

Clients connect to http://127.0.0.1:8000/mcp with the request header Authorization: Bearer your-token. For production, place it behind a reverse proxy (Nginx / Caddy) and enable TLS; do not expose a plaintext public port directly.

Data Directory and Git Sync

The data root directory is specified by the environment variable JD_DATA_DIR, defaulting to <repo root>/data (gitignored):

sync_data / jdcollector sync decision logic:

• Data directory is not a git repository → local save only, skip git;

• Data directory is nested inside another repository (e.g., the tool repository's own ./data) → skip git to prevent data from being committed into a public repository;

• Data directory is a standalone git repository → add → commit → pull --rebase → push.

Example: putting data in a private GitHub repository

set JD_DATA_DIR=D:\jd-data
cd D:\jd-data && git init && git remote add origin git@github.com:you/jd-data.git
# 推送时可用
set JD_GIT_TOKEN=ghp_xxx

LLM Multi-Vendor Refinement

The llm: section of config/settings.yaml:

llm:
  enabled: true        # 关闭即纯规则分析
  provider: deepseek   # 选供应商

Capability ladder: a unified LlmClient abstraction automatically selects the structured output form based on vendor capability — L3 uses each vendor's native strict schema mode (Claude output_config / OpenAI response_format.json_schema / Gemini response_schema), L2 uses JSON mode (DeepSeek / Qwen's response_format.json_object, Ollama's format:"json"), L1 is pure prompt fallback; automatic downgrade on failure of the current tier. Output is uniformly validated via Pydantic; on missing / invalid output, one retry with error feedback is attempted, and if it still fails, it falls back to rule-based results without interrupting the pipeline.

When provider: claude, the top-level model / api_key_env still override the presets for backward compatibility; for other vendors, configure under llm.providers.<name> (field meanings are in the file comments).

Project Structure

config/            # YAML 配置：settings / sites / categories / skills
src/jdcollector/
  crawler/         # 各站点爬虫
  analysis/        # 规则分类 + llm_client 能力梯子 + llm 精分
  portrait/        # 岗位群像报告
  view/            # 静态 HTML 报告站
  mcp_server.py    # MCP 服务器（工具注册 + CLI 入口）
  sync_github.py   # 数据目录 git 同步（独立仓库保护）
tests/             # pytest 单元测试

Disclaimer

⚠️ Please read the following terms carefully before using this project.

1. Usage restriction: This project is for personal learning, data analysis, and technical exchange only. It does not constitute any commercial use, nor is it a basis for job-seeking / hiring decisions.

2. Data is self-provided and self-responsible: Crawling happens on your own machine, and all data compliance responsibility lies with the user. Please comply with the target website's terms of service and robots.txt, as well as local laws and regulations (e.g., the Personal Information Protection Law, the Data Security Law, the Anti-Unfair Competition Law).

3. Crawling boundaries: Only crawl public pages; do not crawl login-required or non-public personal information; do not crack CAPTCHAs, bypass WAF / bans, use proxy pools or large-scale collection, or interfere with the normal service of target websites in any way. Stop when blocked, and never escalate adversarial measures.

4. Data accuracy: JD content is public information on the site and may be outdated or inaccurate; the author does not guarantee its truthfulness, completeness, or usability; the built-in seed sample data is only for validating the full pipeline and does not represent any real job postings.

5. AI output is for reference only: LLM refinement / portrait conclusions are generated by third-party models and may contain bias; please manually review before use.

6. Use at your own risk: This project is open-sourced under the MIT License. The author is not liable for any direct or indirect losses, data leaks, or legal risks arising from the use of this project.

License

MIT © 2026 NaoYun777