Research hub crawler (personal-development knowledge base)
hub_crawlCrawls 11 curated sources into a local SQLite full-text research hub, respecting robots.txt and off-hours windows. Supports dry-run, source limits, and force mode.
Instructions
Politely crawl the 11 curated sources (Farnam Street, LessWrong, Derek Sivers, Julian Shapiro, Internet Archive, Open Library, Project Gutenberg, Wikibooks, PhilArchive, PsyArXiv, CORE) into the local SQLite+FTS5 research hub. Respects robots.txt and each source's off-hours crawl window unless force=true.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| force | No | Bypass the off-hours crawl window (default false). | |
| dbPath | No | Alternative SQLite path (default data/research-hub.sqlite). | |
| dryRun | No | Preview robots.txt + discovery only, write nothing (default false). | |
| sources | No | Source ids to limit the crawl to (default: all 11). | |
| maxItems | No | Max items to discover per source (default 12). | |
| minWords | No | Minimum word count to keep a fetched page (default 60). |