niche-librarian

Give your AI assistant a librarian for one niche: a local search index of the docs and forums that practitioners actually use, served to the agent over MCP.

General web search and a model's own memory are weakest exactly where specialist work lives: version-specific details, error messages, "this broke after the upgrade", and the difference between an official fix and a forum guess. niche-librarian crawls a hand-picked set of sources for one topic, through official APIs wherever they exist. It records who said what, on which version, with what standing (vendor docs, staff reply, accepted answer, or just a post), then ranks results with that in mind. Every result is citable back to the original post or page.

This repo ships the recipe (code, configs and source lists), never crawled content. You run the crawl yourself, and the data stays on your machine in data/, which git ignores.

Topic packs included

Each pack is independent: its own database, its own MCP server, its own ranking. Writing a pack for your own niche is the point. See Writing your own pack.

Related MCP server: loci-mcp

Quick start (home server pack)

You need Python 3.11+ and Ollama, which provides the local embeddings for semantic search. Without Ollama, search still works in keyword-only mode.

1. Install (Linux / macOS):

git clone https://github.com/Nullislander0/niche-librarian.git
cd niche-librarian
python3 -m venv .venv
.venv/bin/pip install -e ".[dev,rerank]"
ollama pull nomic-embed-text
.venv/bin/pytest            # optional: should pass

On Windows (PowerShell), use py -3 -m venv .venv and .venv\Scripts\ in place of .venv/bin/ throughout.

2. Pick the pack and check its sources:

export LIBRARIAN_PACK=homelab     # or pass --pack homelab to each command
.venv/bin/librarian sources       # how each source is accessed, and when that was checked

3. Crawl a sample first. --sample 50 stops each source after 50 threads or pages: a walk around the block that proves every source works, in about 15 minutes. Run the full crawl once you're happy with it.

.venv/bin/librarian crawl --sample 50   # quick end-to-end test of the pack
.venv/bin/librarian crawl               # full first crawl (see "What to expect" below)

4. Search from the terminal, to see what you got:

.venv/bin/librarian outline
.venv/bin/librarian search "Proxmox VE 8 GPU passthrough to VM not working" --explain

5. Connect your AI assistant, as covered under Connect an agent (MCP) below. Then ask your assistant real questions. It searches the index, reads the best threads and cites them.

What to expect from a first crawl

• Time: the docs and wikis take an hour or two. The forums take most of a night, because the crawler paces itself (min_interval_seconds in the pack's sources.yaml). Leave it running. It keeps the machine awake while it works, and it resumes where it stopped if interrupted.

• Disk: packs end up in the hundreds of megabytes up to about 1 GB each, plus a raw response cache in data/cache/.

• Later runs only fetch what changed: new topics, edited wiki pages, new posts in threads you already have.

How it works

sources ──► parsers ──► version tagging ──► chunks ──► SQLite (FTS5 + sqlite-vec) ──► rerank ──► MCP
            (per site)   (versions.yaml)                (one .db per pack)            (ranking.yaml)

• The engine (engine/) is generic: a polite fetcher, parsers, storage, hybrid search, ranking and the MCP server.

• A topic pack (packs/<name>/) is configuration: the source list, version patterns, synonyms, ranking weights and any site-specific parser overrides. To add a topic, write a pack. The engine shouldn't need changes.

Search combines keyword search (SQLite FTS5, BM25) with semantic search (Ollama nomic-embed-text vectors in sqlite-vec), merged with reciprocal rank fusion. A small cross-encoder then re-reads the top candidates against the query (FastEmbed, ONNX Runtime, no PyTorch). On the Esri pack's 200-question test, that raised top-1 accuracy from 73% to about 80%. Last, results are reranked using the pack's signals:

• Version match with the version being asked about, e.g. pve 8.2 or truenas 24.10.

• Recency, only when the question is version-sensitive.

• Authority: vendor docs, knowledge base and bug reports outrank forum opinion.

• Accepted answers, vendor staff, recognized experts, and votes or likes.

• First-hand evidence: logs, configs, "after upgrading…", "this fixed it".

• Author reputation and account age, where the site provides them.

These signals scale relevance rather than replace it, so a popular answer to a different question can't outrank the right one. There's deliberately no backlink weighting, no keyword-stuffing reward and no blanket freshness bias.

Setup details

Platforms. It's developed on Linux, and the tests run on Linux, Windows and macOS in CI.

• Sleep: long jobs (crawl, rebuild, embed, eval) keep the machine awake while they run, and the machine sleeps normally afterwards. On Linux this uses a desktop idle inhibit plus systemd-inhibit, on Windows SetThreadExecutionState, and on macOS caffeinate. Set LIBRARIAN_NO_STAY_AWAKE=1 to turn this off.

• macOS: the usual macOS Pythons (python.org's installer, Apple's, GitHub's) can't load SQLite extensions, which semantic search needs. On macOS the install therefore adds sqlean.py, a drop-in SQLite build that can, and the engine switches to it automatically: nothing to configure, and any Python works. If it ever can't load either, the index runs keyword-only (full-text search plus the reranker) and logs a warning. CI checks that vector search loads on Linux, Windows and macOS.

Settings come from environment variables:

Keep keys in your shell environment, never in the repo.

Everyday commands

librarian crawl [--source ID] [--sample N] [--max-requests N]   # fetch and index; resumable
librarian check                                    # sources.yaml typos: unknown keys and options
librarian sources                                  # sources and their access records
librarian outline                                  # what the index covers: sources, boards, tags, versions
librarian search "query" [--version 11.3] [--explain]
librarian fetch "<chunk_id>" [--scope chunk|document|thread] [--max-tokens N]
librarian rebuild                                  # re-parse saved responses (no network)
librarian retag                                    # reapply versions.yaml to the whole index (seconds)
librarian eval-build / eval                        # build and run the pack's retrieval test

Every raw response is kept in data/cache/, so rebuild can re-parse everything after you change a parser or the chunking, without touching the network.

Each crawl also checks what it parsed. If most documents from a source come out empty, untitled, undated or authorless, or most threads share one title (a site banner instead of the thread's name), the crawl report lists parse_warnings: usually a sign the site's layout changed and its parser needs a look.

Saved searches and alerts

Save a query once, and get only what's new since you last checked:

librarian watch add pve-upgrades "Proxmox VE upgrade problems" --min-score 0.6
librarian crawl --watch --notify        # crawl, then report new matches (desktop notice)
librarian watch run                     # or on its own; writes a report to data/watches/
librarian watch list | remove NAME
librarian search "query" --new-since 2026-11-01     # ad hoc: only recently indexed material

Each document records when it first entered the index, so "new" means genuinely new, not re-crawled. Watches are personal and live in data/watches/, outside the repo. Agents can ask the same thing through MCP: search(..., new_since="YYYY-MM-DD").

Web watches run a query against the live web instead of your index, catching rare mentions on sites you don't crawl:

export BRAVE_API_KEY=...                # https://api-dashboard.search.brave.com
librarian watch add vfio "GPU passthrough regression" --web --freshness pw --site forum.proxmox.com
librarian watch run --web-only

Release alerts use Brave's news index: list your products under web.releases in the pack's sources.yaml, and get one watch per product for new versions, patches, security advisories and end-of-support notices:

web:
  releases:
    - {name: Proxmox VE, match: 'Proxmox'}      # match: must appear in the title or snippet
    - {name: TrueNAS, match: 'TrueNAS'}

librarian watch add-releases            # creates/updates release-proxmox-ve, release-truenas
librarian watch run --web-only --notify
librarian watch add fw "OPNsense firmware" --web --backend brave-news --require 'OPNsense'

Brave's API terms don't allow storing results, so web results are shown, never saved. Only fingerprints of URLs already seen are kept, so later runs show only new links. Your plan's monthly limit is read from Brave's response headers (see the quota line below).

Beyond the index: web search and single pages

Your index holds what you could crawl. For the rest (very recent posts, sites that block crawlers, one-off pages) the agent gets three more MCP tools:

Nothing an agent finds goes into your index until you approve it:

librarian pages list                    # what agents proposed, and why
librarian pages approve [URL ...]       # approve (all pending by default) and index
librarian pages reject [URL ...]        # won't be proposed again
librarian pages add URL --note "..."    # add one yourself
librarian pages remove URL
librarian web "query" --site reddit_gis # search the web yourself; prints the quota

Approved pages become the web_pages source, ranked below the vendor docs and forums your recipe crawls. Pages under a part of a site one of your sources already crawls are refused, because that source's parser keeps richer metadata. What's stored comes from the page's own site, never from search results.

Each pack can name site shortcuts and hosts never to fetch in sources.yaml:

web:
  sites: {reddit_homelab: reddit.com/r/homelab, reddit_proxmox: reddit.com/r/Proxmox}
  blocked_hosts: [example-forum.com]     # e.g. sits behind a bot challenge

Quota: agent searches stop when the month's remaining Brave quota falls below a quarter, so your watches always have room (LIBRARIAN_WEB_AGENT_SHARE=0.75 to change it), and at 50 per server session (LIBRARIAN_WEB_SESSION_CAP). Keys can live in ~/.config/niche-librarian/env (export NAME=value lines): the engine reads that file itself, so MCP clients that don't start from your shell still find them.

Trend signals

For "how much is X being talked about, and about what?" rather than "what's the answer?", a pack can list signals: in its sources.yaml: terms to track and categories to sort mentions into, each with a weight.

signals:
  terms:
    - {name: proxmox-backup, match: 'proxmox backup server|\bpbs\b'}
  categories:
    data_loss: {weight: 1.0, description: "lost or corrupted backups", keywords: 'corrupt|lost data'}
    pricing:   {weight: 0.5, description: "subscription and licensing cost", keywords: 'subscription|price'}

librarian signals run                   # label new documents (local LLM via Ollama, or keywords)
librarian signals report --by month     # counts per term, category and period

Labeling runs on your machine (qwen3:4b by default; --keywords-only without Ollama). Only labels are stored: no text and no authors. Agents get the same table from the MCP signals tool.

Sources whose documents carry numbers (source_type: data, with extra.metrics) appear as measures beside the counts instead of being labeled. Two parsers produce them, both for public-domain US government data:

• openfda: FDA adverse event reports per drug per month, drug shortages and recalls

• nadac: CMS NADAC, the weekly price pharmacies pay per unit, and each week's price changes

• medicaid_sdud: Medicaid outpatient prescriptions per drug per quarter (CMS State Drug Utilization Data), nationally or by state

• partd_formulary: Medicare Part D coverage per quarterly release: share of plans covering a drug and how many require prior authorization or step therapy. CMS ships this as a 2.5 GB ZIP; the parser reads only the two files it needs (~9 MB) with HTTP range requests

Packs about people: pseudonymized authors

A pack can set privacy: {authors: pseudonymize}. Each author is then stored as a stable pseudonym (P-7f3a91, a keyed hash: the same person every week, the handle never stored), handles are scrubbed from quotes, @mentions, greetings and attributions in the text, emails and phone numbers are removed, and profile numbers are coarsened. The signals report then counts posters as well as posts, and shows switch flows: posters who mentioned one term and later another. librarian privacy apply converts documents already in the index; librarian rebuild re-scrubs from the cache with every handle known. librarian privacy audit checks the index for anything left (handles, profile links, emails, phone numbers) without printing the handles; it also runs after every crawl of a pseudonymized pack. The key is created in data/secrets/ (back it up: a new key means new pseudonyms). Pseudonymized isn't anonymous: post text and URLs can lead back to the original post, and the local page cache keeps the raw pages for rebuilds.

Connect an agent (MCP)

Each pack is its own MCP server (librarian serve --pack <pack>), so register one per topic you want an assistant to use. Use absolute paths.

Claude Code

# Linux / macOS
claude mcp add homelab-librarian --scope user -- /ABS/PATH/.venv/bin/librarian serve --pack homelab
# Windows
claude mcp add homelab-librarian --scope user -- C:\ABS\PATH\.venv\Scripts\librarian.exe serve --pack homelab

Then start a new session. To make the assistant reach for it without being asked, add a line like this to your CLAUDE.md: "For home server, Proxmox, TrueNAS or Docker questions, search the homelab-librarian MCP before answering, and cite chunk IDs."

Claude Desktop. Edit claude_desktop_config.json (Settings → Developer → Edit Config). The file lives in ~/Library/Application Support/Claude/ on macOS, %APPDATA%\Claude\ on Windows, and ~/.config/Claude/ on Linux.

{
  "mcpServers": {
    "homelab-librarian": {
      "command": "/ABS/PATH/.venv/bin/librarian",
      "args": ["serve", "--pack", "homelab"]
    }
  }
}

On Windows, use "command": "C:\\ABS\\PATH\\.venv\\Scripts\\librarian.exe". JSON needs the backslashes doubled.

Tools the agent gets:

The tool descriptions coach the agent: one topic per search, pass the user's version, stop after a few empty searches, and say when sources disagree.

Private packs and reports

Packs are looked up in two places, first match wins: your private packs folder, ~/.config/niche-librarian/packs/, then the repo's packs/. Keep any recipe you don't want published (client topics, internal sources) in the private folder: it lives outside the repo, so it can't be committed or pushed. A private pack with the same name as a repo pack replaces it locally. Set LIBRARIAN_PACKS to a :-separated list of folders (; on Windows) to change the search path.

Reports and other results belong under data/ (e.g. data/reports/<pack>/), which git ignores. Two guards back this up: tools/pre-push is a git hook that refuses a push containing reports, data, secrets or a pack that also exists in your private folder (install it with cp tools/pre-push .git/hooks/pre-push), and a test fails in CI if any such file is tracked.

For reports, librarian memo-data (also the MCP memo_data tool) computes every number once: latest and previous value, change, periods and the document to cite, with incomplete FDA months left out, so a writing agent never does arithmetic.

Writing your own pack

A pack is a folder under packs/ holding a few YAML files. Copy packs/homelab/ as a starting point. The full field guide, covering access rules, scoping and platform quirks, is docs/SOURCING.md.

Building a recipe, step by step

1. Pick a narrow niche. "Home servers" or "one product's administration" works; "all of tech" doesn't. Write one sentence on who the pack is for. It becomes the pack's description, which agents read.

2. List where practitioners actually go: vendor docs, the vendor's community, the independent forum everyone recommends, the relevant Stack Exchange tags, a wiki. Ask someone who does the work.

3. Check each site's access before writing any config: one plain request (real content or a block?), whether there's an API or sitemap, and its rough size. See the checklist. Drop anything that blocks or challenges the crawler.

4. Match each source to a parser (table below). If one fits, the source is config only. If none does, write a parser (see below) and test it against a hand-written fixture.

5. Scope each source: categories or tags, a since: date, slug filters, caps for megathreads. Estimate requests × pacing before you run anything.

6. Write versions.yaml and synonyms.yaml for the niche's version numbers, codenames and abbreviations. Check them with librarian retag and a few searches.

7. Run librarian crawl --sample 50. About 15 minutes proves every source works end to end. Read a few results with librarian search ... --explain and librarian fetch.

8. Measure, then tune. If a source has accepted answers, build a retrieval test (eval-build, eval) and change ranking or synonyms only when the numbers improve.

9. Run the full crawl, then share the recipe. The repo never includes the data.

Parsers already available. If your sources run on one of these, a new pack is config only:

Record how you checked access. Every source gets an access: block. A source whose verdict is blocked or needs-key is never crawled:

access: {checked: 2026-11-05, robots: allowed, method: Discourse JSON API, auth: none, verdict: ok}

Before adding a site, check whether a plain request gets real content or a bot challenge, and whether you've accepted any terms that cover it. Prefer an official API or export. If a site challenges or blocks the crawler, mark it blocked and move on; never work around it.

Test your pack's search quality. librarian eval-build picks threads that have accepted answers and rewrites each question in different words. By default it uses a local Ollama model (LIBRARIAN_EVAL_MODEL), which should be one that answers directly rather than "thinking" at length; --rewrite none leaves the queries blank so you can write them yourself. librarian eval then reports how often search re-finds the original thread (top-1, top-5, top-10). Run it before and after any ranking or synonym change. A pack needs a source with accepted answers for this.

A new kind of site needs a parser

Copy engine/parsers/stackexchange.py, the most thoroughly commented example, or the closest existing parser:

# engine/parsers/mysite.py  (or packs/<pack>/parsers/mysite.py for a pack-only parser)
from engine.parsers.base import CrawlContext, Document

class Parser:
    POSITION_KEY_PREFIXES = ("cursor",)          # state keys that hold crawl position

    def __init__(self, ctx: CrawlContext):
        self.ctx = ctx                            # ctx.options = this source's options

    def crawl(self):
        st = self.ctx.state.get("cursor") or {"page": 1}   # resumable position
        while True:
            self.ctx.check_budget()               # respects --max-requests
            data = self.ctx.fetcher.get_json("https://api.example.org/posts",
                                             {"page": st["page"]})   # pacing/cache/UA/blocks
            for post in data["posts"]:
                yield Document(
                    doc_id=f"{self.ctx.source.id}:{post['id']}",     # stable forever
                    source=self.ctx.source.id, source_type=self.ctx.source.source_type,
                    kind="post", url=post["url"], title=post["title"],
                    content_md=post["markdown"], post_date=post["created"],
                    author=post["author"], score=post.get("votes"),
                )
            if not data["has_more"]:
                break
            st["page"] += 1
            self.ctx.state.set("cursor", st)      # save progress after each page

Rules for parsers:

• Make every request through ctx.fetcher, which handles pacing, caching, block detection and the user agent.

• doc_id must be stable. Chunk IDs (<doc_id>#c<n>) are what agents cite.

• Return content as markdown, plus whatever metadata the site really provides (votes, accepted answers, staff status, reputation). Leave a field None rather than guess.

• Replies set parent_id to the thread root. That makes fetch(scope="thread") work, and replies inherit the thread's version tags.

• Parsers don't do version tagging, chunking or ranking. The engine handles those from pack config.

• Add a test under tests/ with a hand-written fixture for the cases you care about, and a structure-only fixture from real pages: after a small crawl, run python tools/make_fixture.py PACK SOURCE NAME. It replays the parser against your cache and keeps the real markup, JSON keys, numbers and dates, but replaces every word, username, host and API key with stand-ins (through a mapping that's never saved, so it can't be undone). tests/test_real_fixtures.py then checks the parser still reads that real structure.

Rules this project follows

• Polite, honest access. It paces requests (including any robots.txt Crawl-delay) and identifies itself with an honest user agent. It honours the agreements you actually accept, such as API terms; robots.txt Disallow rules aren't legally binding in the US and are enforced only for EU/EEA sites (configurable; see docs/SOURCING.md).

• It never works around a block: no bypassing bot challenges, CAPTCHAs or login walls. A blocked source is reported and skipped.

• Recipe, not data. Crawled content is never committed or redistributed.

• Nothing private in the repo: no client data, hostnames, internal URLs or credentials, in code, configs or fixtures.

The reasoning behind these rules, the access-check checklist, scoping advice and per-platform notes are in docs/SOURCING.md.

Layout

engine/
  crawl/fetcher.py      polite HTTP: pacing, cache, retries, block detection
  parsers/              base.py, stackexchange (worked example), vanilla, discourse, mediawiki,
                        sitemap_html, xenforo1, html_md (HTML -> markdown)
  index/                store (FTS5 + sqlite-vec), chunking, Ollama embeddings
  rank/                 rerank.py (fusion + pack signals), cross.py (cross-encoder)
  versions.py           version tagging from versions.yaml
  synonyms.py           query expansion from synonyms.yaml
  indexer.py            crawl -> tag -> chunk -> store -> embed
  search.py             hybrid search, fetch, outline (shared by CLI and MCP)
  evaluate.py           retrieval test: build and run
  awake.py              keep the machine awake during long jobs
  mcp_server/server.py  MCP tools
  cli.py                the `librarian` command
packs/
  homelab/  esri_enterprise/
tests/

License

MIT. See LICENSE. This covers the code and configs only: crawled content belongs to its authors and sites, and is never part of this repository.