Skip to main content
Glama
$ prospector profiles
Available profiles:
  - hospital-tech
  - saas-pain

$ prospector sweep hospital-tech        # two-stage scrape → scored SQLite store
$ prospector report hospital-tech       # evidence-bound Markdown report
$ prospector embed                      # vectors for new items ([semantic] extra)
$ prospector semantic-search "is there a tool that tracks this" --limit 5

prospector pulls Reddit content through the public .json endpoints, scores every post and comment against a per-topic "pain" lexicon, stores the lot in SQLite, and renders a report of recurring unmet needs — each one backed by real permalinks and verbatim quotes. Point it at any niche by dropping in a YAML profile; the flagship profile hunts for a piece of tech missing from hospitals that frontline staff wish existed. The same engine runs as an MCP server, turning Claude into a Reddit research specialist that collects once and reasons over the store many times.

The engine is deterministic plumbing — no LLM is required to scrape, score, or store. Insight is the client's job: Claude via MCP, or an optional built-in --analyze report.

✨ Features

  • Reddit .json client — listings, in-sub search, and comment trees; descriptive User-Agent, 429 Retry-After backoff, on-disk response cache. Optional free OAuth (env vars) lifts the rate limit ~10×.

  • RSS fallback: when .json returns HTTP 403, the client reads the public Atom feeds on www.reddit.com for the rest of the run. It keeps at least 20 seconds between requests and obeys the x-ratelimit-* headers. After a refused request it waits a full rate window, doubles the wait for each refusal in a row (up to 10 minutes), and stops after 6 refusals in a row. Each sweep saves its request counts (ok, 403, 429, other) in the store. On 2026-10-04 the measured budget from a home IP was one request per minute. Feeds carry no vote score and no comment count, so those fields are 0 on this path. On RSS the sweep reads the comments of at least 10 threads per profile (rss_comment_threads in the profile, or --rss-comment-threads), chosen by pain score and then by recency.

  • Semantic search and clusters (optional [semantic] extra): BAAI/bge-small-en-v1.5 vectors (fastembed, CPU) in a sqlite-vec table in the same SQLite file. semantic-search finds posts and comments by meaning. clusters groups items by meaning and shows each group's size, keywords and nearest quotes. Every hit and every example is a stored item with its permalink and a verbatim quote.

  • Retention and deletion: prune deletes items posted more than 7 days ago (configurable), items that now read [deleted] or [removed], and items that Reddit returns as deleted or no longer has (/api/info, 100 ids per request). Their lexicon matches and vectors go too, and old response-cache files are purged.

  • Two-stage scrape — a broad, cheap post sweep, then comment trees fetched only for threads that clear the pain threshold or run hot. Spends the rate-limit budget where the signal is.

  • Deterministic pain scorer — a per-profile weighted-regex lexicon gives every item a transparent pain_score plus the exact patterns that fired. No model, fully reproducible.

  • Evidence-bound reports — a "gap" is structurally dropped unless it clears the profile's thresholds (≥N distinct items, across ≥M subreddits, from ≥K authors), each with a stored permalink + quote. The renderer cannot emit an unbacked claim.

  • Plug-and-play profiles — a topic is one YAML file (subreddits, search_terms, pain_lexicon, thresholds). Swap the niche with zero code changes.

  • MCP server (FastMCP) — 11 tools (reddit_sweep, reddit_search, reddit_fetch_thread, reddit_query, reddit_get_evidence, reddit_stats, reddit_export, reddit_profiles, reddit_profile_get, reddit_semantic_search, reddit_clusters) so Claude can drive the whole loop.

  • Optional standalone analysis — report --analyze adds a one-paragraph thesis per gap via any OpenAI-compatible endpoint, constrained to the fetched evidence. Degrades to stats-only if no key is set.

Related MCP server: slopweaver

🛠 Stack

Python · httpx · Typer · SQLite · PyYAML · FastMCP · (optional) fastembed + sqlite-vec · (optional) any OpenAI-compatible LLM

🚀 Run

pipx install prospector-reddit          # or: uvx prospector-reddit ...
# from source:
pip install -e ".[dev,analyze]"

prospector profiles                     # list topic profiles
prospector sweep hospital-tech          # collect + score + store
prospector query hospital-tech --min-pain 4 --sort pain
prospector report hospital-tech --out reports/hospital.md
prospector report hospital-tech --analyze   # + LLM thesis (needs an LLM endpoint)

# search by meaning (optional extra; the model downloads once, about 67 MB)
pip install -e ".[semantic]"
prospector sweep saas-pain --time week --combine-terms   # one OR search per sub
prospector embed                                         # vectors for new/changed items
prospector semantic-search "is there a tool that tracks this" --limit 5
prospector semantic-search "QUERY" --db PATH --limit 3 --json   # stable JSON contract
prospector clusters --sub selfhosted --examples 2
prospector prune --max-age-days 7                       # age + deleted-on-Reddit

semantic-search --json prints a JSON array. Each hit has exactly the keys permalink, subreddit, title, quote, score and created_utc. score is the cosine similarity to the query (higher is closer), not the Reddit vote score. An empty or missing store prints [] and exits 0.

Daily run on a server

deploy/prospector-daily.sh runs sweep for each profile, then embed, then prune, then one search check, and prints daily: fetched=N embedded=N pruned=N searched=N profiles_ok=N profiles_failed=N failures=N. A profile fails when its sweep fetches nothing or gets more refused requests than good ones. The script exits non-zero when every profile fails or a later step fails. The prune step runs even when a sweep fails. At one RSS request per minute, one profile with 14 subreddits takes about 28 minutes for posts plus about 10 minutes for 10 comment threads. deploy/install-systemd.sh installs it as the user units prospector-sweep.service and prospector-sweep.timer (daily at 09:20, with MemoryMax=2G). Keep the store outside the checkout (default ~/prospector-data). Machine-local settings, such as PROSPECTOR_SWEEP_PROFILES, go in ~/prospector-data/prospector.env.

Higher throughput (optional, free): create a Reddit "script" app and export REDDIT_CLIENT_ID / REDDIT_CLIENT_SECRET before sweeping — the client switches to OAuth (100 req/min). LLM analysis reads FREELLMAPI_BASE_URL+FREELLMAPI_KEY (or the OPENAI_* equivalents).

Use it from Claude (MCP)

Register the server in your MCP client (.mcp.json):

{ "mcpServers": { "prospector": { "command": "prospector", "args": ["mcp"] } } }

Then Claude can reddit_sweep a profile, reddit_query the store, drill hot threads with reddit_fetch_thread, and resolve citations with reddit_get_evidence — collect once, reason many.

🧠 How it works

 profiles/*.yaml ─┐
                  ▼
   RedditClient ──► two-stage scrape ──► lexicon scorer ──► SQLite store
   (.json/OAuth)      posts→comments        pain_score          │
                                                                ▼
                              evidence-bound renderer ◄── Claude (MCP)  or  --analyze
                              (drops under-evidenced gaps)

The core engine never invents anything — it only surfaces what it actually fetched, and the report renderer enforces the evidence contract, so every claimed gap is traceable to real Reddit permalinks and quotes.

🗺 Roadmap

Code complete and verified locally — 163 offline unit tests pass, all modules import, the CLI and the full two-stage sweep run end to end, and all 11 MCP tools register. Built with a frozen interface contract (INTERFACES.md) so the modules integrate cleanly.

  • Known limitation — Reddit blocks datacenter/VPN IPs. Unauthenticated .json (and even OAuth) returns 403 from VPN/hosting-provider IP ranges. On 2026-10-04 .json also returned 403 from a home IP, while the RSS feeds returned 200. The default auto transport then uses RSS. The RSS budget is small (about one request per minute), so a daily sweep of three profiles takes hours.

  • Known limitation: Reddit Data API terms. Reddit says traffic without OAuth can be blocked at any time, and it asks builders to register under its Responsible Builder Policy. This tool is for personal, non-commercial research. It uses a pretrained embedding model and trains nothing on Reddit content. Keep the store private: never commit it.

  • Known limitation — results are hypotheses to validate, not validated needs. Reddit is not ground truth and venting is not a market; the medical profile makes no clinical claim.

  • Generate a real flagship hospital-tech report (pending a live sweep from a clean IP).

  • Semantic search and clusters over stored posts and comments ([semantic] extra).

  • Trend deltas — surface gaps that are rising over time.

📄 License

MIT — see LICENSE. Read-only and non-commercial by design; respects Reddit's terms, no bulk-data redistribution.

Available Tools

9 tools
reddit_exportB

Export stored items as a serialized string in json, csv or md.

Applies the same profile / min_pain filters as reddit_query and returns the rendered payload as text (highest pain first). Useful for handing a dataset back to the user.

Inputs: limit, format, profile, min_pain. Returns: result

reddit_fetch_threadB

Fetch a post's comment tree live and store the comments.

post_id may be a bare id (abc123) or a fullname (t3_abc123). Returns {"post": <post dict or None>, "comments": [<comment dicts>], "comments_collected": N}. The post object is included when it is already in the store (e.g. from a prior sweep/search). Comments are stored so you can cite them as evidence afterwards.

Inputs: post_id (required), min_score, max_comments. Returns: object

reddit_get_evidenceA

Resolve stored items into citable evidence for the given ids.

Returns one EvidenceItem dict per known id (in the order requested) with the permalink and a trimmed verbatim quote — use this to ground any claim in a real Reddit link instead of paraphrasing. Unknown ids are skipped.

Inputs: ids (required). Returns: result

reddit_profile_getA

Return one profile's full configuration as a dict.

Includes its subreddits, search_terms, pain_lexicon (weighted regex rules), pain_threshold and the evidence thresholds that gate reported gaps.

Inputs: name (required). Returns: object

reddit_profilesA

List the available topic profiles by name (e.g. hospital-tech).

A profile defines which subreddits to mine, the search terms, and the pain lexicon. Use this first to discover what can be swept or queried.

Inputs: no named parameters. Returns: result

reddit_queryA

Query already-stored items (no network).

Filters: profile (which sweep collected it), subreddit, min_pain (lower bound on the deterministic pain score), contains (case-insensitive substring over title+body). sort is one of pain|score|new|comments. Returns hydrated item dicts including their matched lexicon patterns.

Inputs: sort, limit, profile, contains, +2 more. Returns: result

reddit_statsA

Summary stats for the store (optionally scoped to one profile).

Returns totals, the post/comment split, a per-subreddit breakdown, the most frequent matched patterns, and the (min, max) created-utc date range.

Inputs: profile. Returns: object

reddit_sweepA

Run a full two-stage sweep for a profile and persist the results.

Stage 1 pulls + scores posts across the profile's subreddits and search terms; stage 2 deep-fetches comments on the highest-pain threads. This makes live Reddit calls and may take a while. time (hour|day|week|month|year|all), limit (posts per sub) and max_threads override the profile defaults when non-empty/non-zero. Returns a SweepResult summary (counts, per-subreddit tallies, top matched patterns).

Inputs: profile (required), time, limit, max_threads. Returns: object

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.8/5.0

    Scored across 9 tools

    Disambiguation5/5

    Each tool serves a distinct function: export, live thread fetch, evidence retrieval, profile configuration, listing profiles, querying stored items, live search, statistics, and full sweep. No two tools have overlapping purposes, ensuring clear selection for an agent.

    Naming Consistency4/5

    All tools share the 'reddit_' prefix and use lowercase with underscores. However, the verb/noun order varies: 'reddit_get_evidence' (verb_noun) contrasts with 'reddit_profile_get' (noun_verb), and 'reddit_profiles' and 'reddit_stats' are noun-only. This minor inconsistency prevents a perfect score.

    Tool Count5/5

    With 9 tools, the server covers the full workflow of discovering profiles, collecting data (search, sweep, fetch_thread), querying stored items, evidence extraction, export, and statistics. The count feels well-scoped for a Reddit mining server.

    Completeness4/5

    The tool set covers core operations from configuration to export. Missing are delete/update operations for stored data or profiles, but these may be intentionally omitted given the read-only mining focus. Minor gap.

    Maintenance

    ActivitySlowing
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers