prospector
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@prospector Search Reddit for pain points in hospital tech"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
$ prospector profiles
Available profiles:
- hospital-tech
- saas-pain
$ prospector sweep hospital-tech # two-stage scrape → scored SQLite store
$ prospector report hospital-tech # evidence-bound Markdown report
$ prospector embed # vectors for new items ([semantic] extra)
$ prospector semantic-search "is there a tool that tracks this" --limit 5prospector pulls Reddit content through the public .json endpoints, scores every
post and comment against a per-topic "pain" lexicon, stores the lot in SQLite, and renders
a report of recurring unmet needs — each one backed by real permalinks and verbatim
quotes. Point it at any niche by dropping in a YAML profile; the flagship profile hunts for
a piece of tech missing from hospitals that frontline staff wish existed. The same engine
runs as an MCP server, turning Claude into a Reddit research specialist that collects once
and reasons over the store many times.
The engine is deterministic plumbing — no LLM is required to scrape, score, or store.
Insight is the client's job: Claude via MCP, or an optional built-in --analyze report.
✨ Features
Reddit
.jsonclient — listings, in-sub search, and comment trees; descriptive User-Agent,429Retry-After backoff, on-disk response cache. Optional free OAuth (env vars) lifts the rate limit ~10×.RSS fallback: when
.jsonreturns HTTP 403, the client reads the public Atom feeds onwww.reddit.comfor the rest of the run. It keeps at least 20 seconds between requests and obeys thex-ratelimit-*headers. After a refused request it waits a full rate window, doubles the wait for each refusal in a row (up to 10 minutes), and stops after 6 refusals in a row. Each sweep saves its request counts (ok, 403, 429, other) in the store. On 2026-10-04 the measured budget from a home IP was one request per minute. Feeds carry no vote score and no comment count, so those fields are 0 on this path. On RSS the sweep reads the comments of at least 10 threads per profile (rss_comment_threadsin the profile, or--rss-comment-threads), chosen by pain score and then by recency.Semantic search and clusters (optional
[semantic]extra):BAAI/bge-small-en-v1.5vectors (fastembed, CPU) in asqlite-vectable in the same SQLite file.semantic-searchfinds posts and comments by meaning.clustersgroups items by meaning and shows each group's size, keywords and nearest quotes. Every hit and every example is a stored item with its permalink and a verbatim quote.Retention and deletion:
prunedeletes items posted more than 7 days ago (configurable), items that now read[deleted]or[removed], and items that Reddit returns as deleted or no longer has (/api/info, 100 ids per request). Their lexicon matches and vectors go too, and old response-cache files are purged.Two-stage scrape — a broad, cheap post sweep, then comment trees fetched only for threads that clear the pain threshold or run hot. Spends the rate-limit budget where the signal is.
Deterministic pain scorer — a per-profile weighted-regex lexicon gives every item a transparent
pain_scoreplus the exact patterns that fired. No model, fully reproducible.Evidence-bound reports — a "gap" is structurally dropped unless it clears the profile's thresholds (≥N distinct items, across ≥M subreddits, from ≥K authors), each with a stored permalink + quote. The renderer cannot emit an unbacked claim.
Plug-and-play profiles — a topic is one YAML file (
subreddits,search_terms,pain_lexicon, thresholds). Swap the niche with zero code changes.MCP server (FastMCP) — 11 tools (
reddit_sweep,reddit_search,reddit_fetch_thread,reddit_query,reddit_get_evidence,reddit_stats,reddit_export,reddit_profiles,reddit_profile_get,reddit_semantic_search,reddit_clusters) so Claude can drive the whole loop.Optional standalone analysis —
report --analyzeadds a one-paragraph thesis per gap via any OpenAI-compatible endpoint, constrained to the fetched evidence. Degrades to stats-only if no key is set.
Related MCP server: slopweaver
🛠 Stack
Python · httpx · Typer · SQLite · PyYAML · FastMCP · (optional) fastembed + sqlite-vec · (optional) any OpenAI-compatible LLM
🚀 Run
pipx install prospector-reddit # or: uvx prospector-reddit ...
# from source:
pip install -e ".[dev,analyze]"
prospector profiles # list topic profiles
prospector sweep hospital-tech # collect + score + store
prospector query hospital-tech --min-pain 4 --sort pain
prospector report hospital-tech --out reports/hospital.md
prospector report hospital-tech --analyze # + LLM thesis (needs an LLM endpoint)
# search by meaning (optional extra; the model downloads once, about 67 MB)
pip install -e ".[semantic]"
prospector sweep saas-pain --time week --combine-terms # one OR search per sub
prospector embed # vectors for new/changed items
prospector semantic-search "is there a tool that tracks this" --limit 5
prospector semantic-search "QUERY" --db PATH --limit 3 --json # stable JSON contract
prospector clusters --sub selfhosted --examples 2
prospector prune --max-age-days 7 # age + deleted-on-Redditsemantic-search --json prints a JSON array. Each hit has exactly the keys
permalink, subreddit, title, quote, score and created_utc. score is the
cosine similarity to the query (higher is closer), not the Reddit vote score. An empty
or missing store prints [] and exits 0.
Daily run on a server
deploy/prospector-daily.sh runs sweep for each profile, then embed, then prune,
then one search check, and prints daily: fetched=N embedded=N pruned=N searched=N profiles_ok=N profiles_failed=N failures=N. A profile fails when its sweep fetches nothing or
gets more refused requests than good ones. The script exits non-zero when every profile
fails or a later step fails. The prune step runs even when a sweep fails. At one RSS request per minute, one profile with 14
subreddits takes about 28 minutes for posts plus about 10 minutes for 10 comment threads. deploy/install-systemd.sh installs it as the
user units prospector-sweep.service and prospector-sweep.timer (daily at 09:20, with
MemoryMax=2G). Keep the store outside the checkout (default ~/prospector-data).
Machine-local settings, such as PROSPECTOR_SWEEP_PROFILES, go in
~/prospector-data/prospector.env.
Higher throughput (optional, free): create a Reddit "script" app and export
REDDIT_CLIENT_ID / REDDIT_CLIENT_SECRET before sweeping — the client switches to
OAuth (100 req/min). LLM analysis reads FREELLMAPI_BASE_URL+FREELLMAPI_KEY (or the
OPENAI_* equivalents).
Use it from Claude (MCP)
Register the server in your MCP client (.mcp.json):
{ "mcpServers": { "prospector": { "command": "prospector", "args": ["mcp"] } } }Then Claude can reddit_sweep a profile, reddit_query the store, drill hot threads with
reddit_fetch_thread, and resolve citations with reddit_get_evidence — collect once,
reason many.
🧠 How it works
profiles/*.yaml ─┐
▼
RedditClient ──► two-stage scrape ──► lexicon scorer ──► SQLite store
(.json/OAuth) posts→comments pain_score │
▼
evidence-bound renderer ◄── Claude (MCP) or --analyze
(drops under-evidenced gaps)The core engine never invents anything — it only surfaces what it actually fetched, and the report renderer enforces the evidence contract, so every claimed gap is traceable to real Reddit permalinks and quotes.
🗺 Roadmap
Code complete and verified locally — 163 offline unit tests pass, all modules import, the CLI and
the full two-stage sweep run end to end, and all 11 MCP tools register. Built with a frozen
interface contract (INTERFACES.md) so the modules integrate cleanly.
Known limitation — Reddit blocks datacenter/VPN IPs. Unauthenticated
.json(and even OAuth) returns403from VPN/hosting-provider IP ranges. On 2026-10-04.jsonalso returned403from a home IP, while the RSS feeds returned200. The defaultautotransport then uses RSS. The RSS budget is small (about one request per minute), so a daily sweep of three profiles takes hours.Known limitation: Reddit Data API terms. Reddit says traffic without OAuth can be blocked at any time, and it asks builders to register under its Responsible Builder Policy. This tool is for personal, non-commercial research. It uses a pretrained embedding model and trains nothing on Reddit content. Keep the store private: never commit it.
Known limitation — results are hypotheses to validate, not validated needs. Reddit is not ground truth and venting is not a market; the medical profile makes no clinical claim.
Generate a real flagship
hospital-techreport (pending a live sweep from a clean IP).Semantic search and clusters over stored posts and comments (
[semantic]extra).Trend deltas — surface gaps that are rising over time.
📄 License
MIT — see LICENSE. Read-only and non-commercial by design; respects Reddit's terms, no bulk-data redistribution.
Available Tools
9 toolsreddit_exportB
Export stored items as a serialized string in json, csv or md.
Applies the same profile / min_pain filters as reddit_query and
returns the rendered payload as text (highest pain first). Useful for handing
a dataset back to the user.
Inputs: limit, format, profile, min_pain. Returns: result
reddit_fetch_threadB
Fetch a post's comment tree live and store the comments.
post_id may be a bare id (abc123) or a fullname (t3_abc123).
Returns {"post": <post dict or None>, "comments": [<comment dicts>], "comments_collected": N}. The post object is included when it is already in
the store (e.g. from a prior sweep/search). Comments are stored so you can
cite them as evidence afterwards.
Inputs: post_id (required), min_score, max_comments. Returns: object
reddit_get_evidenceA
Resolve stored items into citable evidence for the given ids.
Returns one EvidenceItem dict per known id (in the order requested) with the permalink and a trimmed verbatim quote — use this to ground any claim in a real Reddit link instead of paraphrasing. Unknown ids are skipped.
Inputs: ids (required). Returns: result
reddit_profile_getA
Return one profile's full configuration as a dict.
Includes its subreddits, search_terms, pain_lexicon (weighted regex rules), pain_threshold and the evidence thresholds that gate reported gaps.
Inputs: name (required). Returns: object
reddit_profilesA
List the available topic profiles by name (e.g. hospital-tech).
A profile defines which subreddits to mine, the search terms, and the pain lexicon. Use this first to discover what can be swept or queried.
Inputs: no named parameters. Returns: result
reddit_queryA
Query already-stored items (no network).
Filters: profile (which sweep collected it), subreddit, min_pain
(lower bound on the deterministic pain score), contains (case-insensitive
substring over title+body). sort is one of pain|score|new|comments.
Returns hydrated item dicts including their matched lexicon patterns.
Inputs: sort, limit, profile, contains, +2 more. Returns: result
reddit_searchA
Search Reddit live for a query and store the matching posts.
If subreddits is given the search is restricted to each of them (results
merged + de-duplicated); otherwise it is a site-wide search. Posts are stored
so you can drill into them later with reddit_query /
reddit_get_evidence / reddit_fetch_thread. Returns the stored posts as
dicts (id, subreddit, author, score, permalink, title, body, ...).
Inputs: query (required), sort, time, limit, +1 more. Returns: result
reddit_statsA
Summary stats for the store (optionally scoped to one profile).
Returns totals, the post/comment split, a per-subreddit breakdown, the most frequent matched patterns, and the (min, max) created-utc date range.
Inputs: profile. Returns: object
reddit_sweepA
Run a full two-stage sweep for a profile and persist the results.
Stage 1 pulls + scores posts across the profile's subreddits and search
terms; stage 2 deep-fetches comments on the highest-pain threads. This makes
live Reddit calls and may take a while. time (hour|day|week|month|year|all),
limit (posts per sub) and max_threads override the profile defaults
when non-empty/non-zero. Returns a SweepResult summary (counts, per-subreddit
tallies, top matched patterns).
Inputs: profile (required), time, limit, max_threads. Returns: object
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 9 tools
Each tool serves a distinct function: export, live thread fetch, evidence retrieval, profile configuration, listing profiles, querying stored items, live search, statistics, and full sweep. No two tools have overlapping purposes, ensuring clear selection for an agent.
All tools share the 'reddit_' prefix and use lowercase with underscores. However, the verb/noun order varies: 'reddit_get_evidence' (verb_noun) contrasts with 'reddit_profile_get' (noun_verb), and 'reddit_profiles' and 'reddit_stats' are noun-only. This minor inconsistency prevents a perfect score.
With 9 tools, the server covers the full workflow of discovering profiles, collecting data (search, sweep, fetch_thread), querying stored items, evidence extraction, export, and statistics. The count feels well-scoped for a Reddit mining server.
The tool set covers core operations from configuration to export. Missing are delete/update operations for stored data or profiles, but these may be intentionally omitted given the read-only mining focus. Minor gap.
Maintenance
Related MCP Connectors
Hosted Web Crawler MCP server for Claude, ChatGPT and Cursor, with each action switched per account.
Hosted YouTube MCP server for Claude, ChatGPT and Cursor, with each action switched per account.
Hosted X (Twitter) MCP server for Claude, ChatGPT and Cursor, with each action switched per account.
Hosted X Ads MCP server for Claude, ChatGPT and Cursor, with each action switched per account.
Related MCP Servers
- AlicenseAqualityBmaintenanceMCP server to dynamically load Claude Code skills into AI agents515MIT
- AlicenseNot gradedqualityAmaintenanceA local-first MCP server that helps Claude Code prioritize your work by searching across your tools like GitHub and Slack.7 npmMIT
- AlicenseNot gradedqualityDmaintenanceA local MCP server that connects Claude to your YouTube channel, enabling video analysis, performance tracking, comment reading, and niche suggestions.ISC
- AlicenseNot gradedqualityDmaintenanceLocal-first MCP server that gives Claude Code web search, page reading, video transcription, and image analysis — without paid API keys. Runs SearXNG + whisper.cpp natively on Apple Silicon for zero-cost, low-latency research workflows.1MIT