neutrinos-mcp
# neutrinos-mcp
Local AI-first knowledge pipeline over `documentation.neutrinos.com` (ClickHelp portal).
## Layout
- `scraper/scrape.py` — ingestion: sitemap-diff scraper → CSV. Stdlib only.
- `data-source/documentation-source.csv` — the source of truth. One row per topic:
`publication, slug, url, lastmod, title, content_markdown, outgoing_links, fetched_at`
- `data-source/cache/` — per-topic markdown cache (lets the CSV be rebuilt offline).
- `data-source/state.json` — lastmod per topic (change detection).
- `docker/` — container: loops daily at 14:00 IST, catches up on startup if a run
was missed. `restart: unless-stopped` + healthcheck (last_run.json < 48h old).
- `logs/scrape.log` — JSON-lines run log (10 MB × 3 rotation).
- `logs/last_run.json` — last run status; `partial`/`failed` includes per-topic failures.
- `ai-data/` — (next phase) derived SQLite + sqlite-vec + knowledge graph, embedded
locally via an Ollama sidecar (`nomic-embed-text`).
## How scraping works
The ClickHelp reader at `/articles/#!pub/slug` is a login-walled SPA, but
`/article/{pub}/{slug}` serves complete server-rendered HTML with no auth
(verified 2026-09-07). The sitemap index (`/sitemaps/sitemap.xml`) enumerates all
publications → per-publication sitemaps → every topic with `lastmod`. Each run:
1. read sitemaps, diff `lastmod` against `state.json`
2. fetch only changed topics (HTML → markdown-ish via stdlib HTMLParser)
3. rewrite `documentation-source.csv` atomically (cache + state)
4. append a JSON run record to `logs/scrape.log`, update `logs/last_run.json`
Full corpus: 3,141 topics / 53 publications, ~9.2 MB of markdown text.
First fetch took ~2 min at concurrency 8; a no-change run takes ~3 s.
## Commands
python3 scraper/scrape.py # one run (diff + fetch changes)
python3 scraper/scrape.py --full # re-fetch everything
python3 scraper/scrape.py --loop # the container's daily loop
python3 scraper/scrape.py --selftest # assert-based extraction checks
docker compose up -d # start the scheduled scraper
TDQS
Scored across 6 tools
Each tool targets a distinct stage of documentation interaction: discover products, search, fetch content, navigate page relationships, compare versions, and explore the knowledge graph. The only mild overlap is between list_related and traverse_knowledge_graph, but the descriptions clarify page-level navigation versus entity-level thematic exploration.
Tool names follow a consistent imperative snake_case pattern, mostly verb_noun (search_docs, fetch_document, list_products, compare_versions, traverse_knowledge_graph). Minor deviations include list_related lacking an explicit object and search_docs/fetch_document mixing plural and singular noun forms.
Six tools is well-scoped for a documentation knowledge server; each tool covers a distinct need without redundancy. The set is neither sparse nor bloated, and every tool earns its place in the workflow.
The surface covers the full read-oriented documentation workflow: product discovery, search, content retrieval, navigation, cross-version comparison, and conceptual graph exploration. For a read-only documentation MCP, there are no obvious dead ends or missing operations.