Skip to main content
Glama
Kadihx
by Kadihx

Research hub crawler (personal-development knowledge base)

hub_crawl

Crawls 11 curated sources into a local SQLite full-text research hub, respecting robots.txt and off-hours windows. Supports dry-run, source limits, and force mode.

Instructions

Politely crawl the 11 curated sources (Farnam Street, LessWrong, Derek Sivers, Julian Shapiro, Internet Archive, Open Library, Project Gutenberg, Wikibooks, PhilArchive, PsyArXiv, CORE) into the local SQLite+FTS5 research hub. Respects robots.txt and each source's off-hours crawl window unless force=true.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
forceNoBypass the off-hours crawl window (default false).
dbPathNoAlternative SQLite path (default data/research-hub.sqlite).
dryRunNoPreview robots.txt + discovery only, write nothing (default false).
sourcesNoSource ids to limit the crawl to (default: all 11).
maxItemsNoMax items to discover per source (default 12).
minWordsNoMinimum word count to keep a fetched page (default 60).

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.1.0

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the behavioral disclosure burden. It does well by exposing non-obvious behavior: it respects robots.txt, follows each source's off-hours crawl window, and allows a force bypass. It does not detail database write semantics, runtime expectations, or the exact scope of force, but the core side effects are disclosed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single sentence front-loads the core action and constraint, then packs the source list and force caveat without redundancy. Every clause contributes either action, scope, or a behavioral condition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a network-crawling tool with no output schema and no annotations, the description offers enough to select and invoke it: what it crawls, where it writes, and which switch changes scheduling behavior. It does not describe return value shape or warn about long-running/network-heavy execution, but the essential operation is clear.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already explains all six parameters. The description adds a little extra context by tying force=true to the off-hours bypass and listing the exact source ids, but it does not substantially deepen parameter meaning beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb ('crawl'), names the exact resource (11 curated sources), and states the destination ('local SQLite+FTS5 research hub'). It clearly differentiates hub_crawl from sibling read-only tools like hub_query and hub_stats by describing an ingestion/population action.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It provides clear context: use this tool to populate the local research hub from a defined set of external sources, with politeness constraints. It does not explicitly say 'use hub_query for reading' or list when not to use it, so it stops short of full exclusion guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.