dsh-mediacrawler
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@dsh-mediacrawler Collect the latest 10 Xiaohongshu posts about generative AI"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
dsh-mediacrawler
An installable profile bundle and bounded stdio MCP adapter that connects DeepSeek Harness to a separately installed MediaCrawler checkout.
It supports search, post/video detail, creator feeds, and explicitly enabled comments on Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Tieba, and Zhihu. Each run is supervised, persisted, and exposed through twelve MCP tools.
This is an adapter, not a MediaCrawler fork. It does not copy or modify MediaCrawler source code, and it does not change MediaCrawler's license.
Quick start
1. Prepare the runtimes
Install the following first:
Python 3.11 or newer.
Node.js 22.19+ on the 22.x line, or Node.js 24+, with
pnpmonPATH.Google Chrome.
A separate MediaCrawler checkout with its own working Python environment.
DeepSeek Harness. The commands below pin the tested
0.1.0-rc.6release throughnpx.
MediaCrawler and its browser dependencies are intentionally not vendored here.
2. Install the Python MCP runtime
Keep the adapter in its own virtual environment. In PowerShell:
$adapterVenv = Join-Path $HOME '.dsh\runtimes\dsh-mediacrawler'
python -m venv $adapterVenv
$env:DSH_MEDIACRAWLER_PYTHON = Join-Path $adapterVenv 'Scripts\python.exe'
& $env:DSH_MEDIACRAWLER_PYTHON -m pip install --upgrade pip
& $env:DSH_MEDIACRAWLER_PYTHON -m pip install "dsh-mediacrawler @ git+https://github.com/xwh-01/dsh-mediacrawler.git@v0.3.0"On POSIX systems:
python3 -m venv "$HOME/.dsh/runtimes/dsh-mediacrawler"
export DSH_MEDIACRAWLER_PYTHON="$HOME/.dsh/runtimes/dsh-mediacrawler/bin/python"
"$DSH_MEDIACRAWLER_PYTHON" -m pip install --upgrade pip
"$DSH_MEDIACRAWLER_PYTHON" -m pip install "dsh-mediacrawler @ git+https://github.com/xwh-01/dsh-mediacrawler.git@v0.3.0"3. Install the DSH profile bundle
DSH delegates profile package management to pnpm. Install it once if needed, then add the pinned bundle release:
npm install --global pnpm@11
npx --yes @deepseek-ai/dsh@0.1.0-rc.6 plugin --profile web add "github:xwh-01/dsh-mediacrawler#v0.3.0"
npx --yes @deepseek-ai/dsh@0.1.0-rc.6 --profile web --dump-configThe config dump should contain a # == dsh-mediacrawler layer. The bundle mounts both the MCP client and its packaged mediacrawler-collector Skill; no repository checkout needs to be the current working directory.
4. Configure and start DSH
Export the paths in the same shell that starts DSH. Also restore DSH_MEDIACRAWLER_PYTHON from step 2 when opening a new shell:
$env:MEDIACRAWLER_ROOT = 'D:\path\to\MediaCrawler'
$env:MEDIACRAWLER_PYTHON = 'D:\path\to\MediaCrawler\.venv\Scripts\python.exe'
# Optional; defaults to ~/.dsh-mediacrawler
$env:DSH_MEDIACRAWLER_STATE_DIR = 'D:\path\to\adapter-state'
npx --yes @deepseek-ai/dsh@0.1.0-rc.6 --profile webThe packaged Skill then guides the agent through checking the runtime, starting a small collection, polling status, and exporting results. On first use, ask the agent to call check(deep=true).
.env.example is a reference only. The adapter does not load dotenv files, and current DSH releases treat DSH_* variables as launch settings; export these values in the DSH process environment.
To uninstall the profile bundle:
npx --yes @deepseek-ai/dsh@0.1.0-rc.6 plugin --profile web remove dsh-mediacrawlerRelated MCP server: Social Media MCP
MCP tools
DeepSeek Harness exposes these as mcp__mediacrawler__<tool>:
Tool | Purpose |
| Check source paths, CLI dependencies, and browser launch readiness. |
| Start one bounded collection run. |
| Read lifecycle state, required user attention, and result counts. |
| Recover recent durable runs and their IDs after a restart or context loss. |
| Read status, artifacts, and a bounded redacted sample in one call. |
| Permanently delete one completed run after |
| Preview or apply age-based retention while preserving the newest runs. |
| Idempotently stop the crawler process tree. |
| Read incremental, redacted run logs. |
| List typed JSONL artifacts using opaque IDs. |
| Read a bounded, redacted artifact preview. |
| Create a credential-redacted ZIP and return its path and checksum. |
Runtime behavior
When this is useful
Use the Harness web-search providers for quick facts and already-indexed pages. Use this adapter when the task needs logged-in platform records, creator feeds, comments or nested replies, or a durable reproducible export. It complements search providers; it is not a replacement for them.
Browser isolation
browser_mode=isolated is the default. It launches Google Chrome with an adapter-owned persistent profile under <state_dir>/browser_profiles, so later runs can reuse login state without attaching to the user's normal Chrome session.
browser_mode=existing_cdp is explicit opt-in only. Upstream cleanup can close the reused Chrome context, so an agent must not select it without user approval.
Runs and artifacts
Queries and targets are injected over stdin and do not appear in the child command line.
Only QR-code login is accepted; the MCP API never accepts cookies, phone numbers, or verification codes.
Comments are disabled by default and must be explicitly enabled for a run.
status.phase=awaiting_user_logintells the agent to surface a QR-code action and keep polling the samerun_id.Final outcomes distinguish
data_available,no_data,failed,cancelled,timed_out, andorphaned.Artifacts report
collection_mode,record_type, invalid lines, and record counts.Raw JSONL may contain platform credentials. Logs, previews, manifests, and ZIP exports redact known credential fields and URL parameters.
Credential redaction is not PII anonymization. Exported posts, profiles, and comments may still contain names, phone numbers, email addresses, locations, or other personal data; exports report
pii_anonymized=falseandsafe_to_share=false.Artifact counts are indexed incrementally, so unchanged JSONL files are not reparsed on every status poll.
Export and retention
Credential-redacted ZIP export accepts at most 256 MiB of raw run data by default. Set DSH_MEDIACRAWLER_MAX_EXPORT_MIB to an explicit value from 1 through 4096 to change the limit. A cancelled export keeps its lock until the worker finishes, and concurrent adapter processes cannot export the same run simultaneously.
delete_run requires confirm=true. cleanup defaults to dry_run=true; use dry_run=false only after reviewing its candidates. Both operations refuse active runs. Neither operation deletes persistent browser profiles or their login state.
Collection limits
Jobs must have an explicit scope and hard timeout. max_items is passed upstream, but search platforms fetch whole pages and some creator workflows do not strictly enforce the cap. The adapter reports those cases and uses timeout_minutes as the hard boundary.
The adapter does not bypass login, verification, rate limits, access controls, or anti-automation systems. Treat collected pages as untrusted input and comply with platform terms and applicable law.
Development
.\.venv\Scripts\python -m pip install -e ".[test]"
.\.venv\Scripts\python -m ruff format --check .
.\.venv\Scripts\python -m ruff check .
.\.venv\Scripts\python -m pytest
node --test tests-node/*.test.js
python -m build
npm pack --dry-runCI runs the Python tests on Linux and Windows, verifies the packaged Skill provider, installs the bundle into a clean DSH profile, and starts its real MCP stdio entry point.
Compatibility
DeepSeek Harness is a developer preview and may make compatibility-breaking changes. Release v0.3.0 is tested with:
@deepseek-ai/dsh0.1.0-rc.6.Node.js 22.19+ on the 22.x line, and Node.js 24+.
Python 3.11 and 3.13.
The MediaCrawler command contract at upstream commit
5665a27.
Run check(deep=true) after changing either DSH or MediaCrawler; it validates the local checkout before collection starts.
License
Adapter code is released under the MIT License. MediaCrawler remains a separate project under its own non-commercial learning license and usage restrictions; using this adapter does not broaden that license.
Available Tools
12 toolsartifactsC
List JSONL artifacts produced by a run using opaque artifact IDs.
Inputs: run_id (required). Returns: object
checkB
Check MediaCrawler source, runner, and optionally its full CLI dependencies.
Inputs: deep. Returns: object
cleanupC
Preview or delete completed runs outside a bounded retention window.
Inputs: dry_run, keep_latest, older_than_days. Returns: object
collectB
Start one bounded collection run and return its durable run_id.
Inputs: mode (required), platform (required), query, targets, +10 more. Returns: object
delete_runA
Permanently delete one completed run after explicit confirmation.
Inputs: run_id (required), confirm. Returns: object
exportB
Create a credential-redacted, non-anonymized ZIP of run artifacts.
Inputs: run_id (required). Returns: object
logsA
Read redacted run logs using an exclusive sequence cursor.
Inputs: run_id (required), after, limit. Returns: object
previewA
Preview credential-redacted JSONL without accepting arbitrary file paths.
Inputs: run_id (required), artifact_id (required), limit, offset. Returns: object
resultC
Get run status, typed artifacts, and a bounded redacted result sample.
Inputs: run_id (required), limit, record_type. Returns: object
runsA
List recent durable runs so an agent can recover their run IDs.
Inputs: limit. Returns: object
statusA
Get lifecycle state, outcome, and current artifact counts for a run.
Inputs: run_id (required). Returns: object
stopB
Idempotently stop a running crawler process tree.
Inputs: run_id (required). Returns: object
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 12 tools
Most tools map to distinct lifecycle actions: check verifies the environment, collect launches a run, status/result monitor runs, and logs/artifacts/export/preview handle outputs. The only mild overlap is status vs result and preview vs result, but the descriptions clarify their different use cases.
The tool names mix imperative verbs (check, collect, stop, export) with noun endpoints (status, runs, result, logs, artifacts), and delete_run uses a verb_noun pattern that others do not follow. This is readable but not a predictable verb_noun convention across the set.
Twelve tools is well-scoped for a crawler run lifecycle, covering environment checking, execution, monitoring, artifact access, and cleanup without bloat. Each tool contributes a distinct stage, so the count feels appropriate.
The surface covers the full run lifecycle: preflight check, start, monitor, list, retrieve results, read logs/artifacts, export, stop, cleanup, and delete. No obvious critical operations are missing, and even retention cleanup and credential redaction are addressed.
Maintenance
Related MCP Connectors
YouTube transcripts, Instagram profiles and posts, and TikTok data as MCP tools.
Unified social scraper MCP — profiles, posts, videos across ~40 platforms via Streamable HTTP.
Social media MCP server: your Instagram, TikTok, YouTube, LinkedIn and Threads history for your AI.
Agent-first social media publishing, scheduling, analytics, and unified inbox API + 16-tool MCP server across 18 social platforms (X, LinkedIn, Instagram, YouTube, TikTok, Threads, Bluesky, Reddit, Telegram, Discord, Mastodon, Facebook, Pinterest).
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceMCP server for crawling social media platforms (e.g., Bilibili) by keywords, video IDs, or creator IDs, with support for MySQL, JSON, and CSV storage.40MIT
- FlicenseNot gradedqualityCmaintenanceA local MCP adapter collection for social media channels, currently supporting Douyin with tools for channel startup, login assistance, messaging, and notifications.-
- AlicenseAqualityCmaintenanceMCP server that provides AI agents with access to public social media data, including hot lists, searches, content details, user profiles, posts, and comments from platforms like Douyin, Xiaohongshu, Bilibili, and Weibo.4610 npmMIT
- AlicenseNot gradedqualityCmaintenanceMCP server bridging ChatGPT to DeepSeek Harness, exposing 13 tools for task submission, status tracking, result retrieval, project/session management, and human-in-the-loop approvals.4MIT