Skip to main content
Glama

nerve-lit

A personal, semantically searchable literature database for peripheral nerve surgery: about 41,000 PubMed records, open-access full text where it exists, embedded locally and served to an AI assistant as an MCP server.

Built by a plastic surgeon for his own research. Everything runs on one machine; no article text ever leaves it.

Nederlandse versie: README.nl.md.

Pipeline

PubMed (E-utilities)  →  SQLite metadata  →  full text (PMC OA / Unpaywall)  →  chunks
   →  embeddings (bge-m3 via Ollama, 1024-dim)  →  LanceDB  →  kNN + reranker  →  MCP tools
  1. Harvest. A scope query on PubMed (harvester/queries.py) is fetched through the rate-limited E-utilities client and parsed into SQLite (metadata plus a per-article pipeline status). Idempotent and resumable.

  2. Full text, legal routes only. Per article, stop at the first hit: PMC open-access JATS XML (NCBI only serves the body where it holds the rights), then the best open-access PDF via Unpaywall. Anything else is marked needs-institutional and exported as a CSV for manual retrieval through a library subscription. No scraping, no Sci-Hub.

  3. Index. JATS or PDF is split into ~650-token chunks with 20 % overlap on sentence boundaries; reference lists, licence blocks and cover pages are stripped. Articles without full text are indexed on their abstract, so the whole corpus stays findable. Embeddings come from bge-m3 running locally in Ollama; vectors live in LanceDB next to the SQLite metadata.

  4. Search. kNN over LanceDB (top 50), then BAAI/bge-reranker-v2-m3 on the GPU re-ranks to the best 8. About 1.8 s warm. Without CUDA the reranker is skipped instead of taking 45 s on CPU.

  5. Serve. A FastMCP server exposes three read-only tools: search_nerve_lit(query, k, year_from, year_to, full_text_only), get_nerve_lit_article(pmid), get_nerve_lit_stats(). Every hit says whether it came from jats, pdf or abstract.

Related MCP server: NCBI Literature Search MCP Server

Layout

harvester/   queries.py  scope query (~41k articles)
             ncbi.py     rate-limited E-utilities client
             parse.py    PubMed XML -> dicts
             db.py       SQLite: metadata + pipeline status
             harvest.py  phase 1 CLI
             fulltext.py phase 2 CLI (open-access cascade)
             report.py   coverage, samples, status export
index/       extract.py  JATS/PDF -> chunks
             embed.py    Ollama bge-m3
             store.py    LanceDB
             build.py    phase 3 CLI
             search.py   kNN + optional reranker
mcp-server/  server.py   phase 4: MCP tools
tests/                   parsing, chunking, tool-name contract
data/                    SQLite + LanceDB (gitignored, rebuildable)

The corpus (PDFs, XML, index) is never committed: it is rebuildable from the pipeline and lives in a synced folder outside the repo.

Setup

Requires Python 3.12+, uv, Ollama and, for the reranker, an NVIDIA GPU with CUDA.

uv sync --extra index --extra mcp     # base + PDF/LanceDB + MCP server
uv sync --extra rerank                # optional: reranker (~2.5 GB torch, CUDA wheels)
copy .env.example .env                # then fill it in
ollama pull bge-m3

.env needs at least NCBI_EMAIL, UNPAYWALL_EMAIL and PDF_DIR. An NCBI API key is free and raises the rate limit from 3 to 10 requests per second. Both NCBI and Unpaywall ask callers to identify themselves with an e-mail address.

Usage

# Phase 1: metadata
uv run python -m harvester.harvest count --query scope
uv run python -m harvester.harvest run   --query scope
uv run python -m harvester.harvest stats

# Phase 2: full text through open-access routes
uv run python -m harvester.fulltext fetch
uv run python -m harvester.fulltext export        # -> needs-institutional.csv

# Phase 3: index
uv run python -m index.build run
uv run python -m index.build run --retry-failed
uv run python -m index.build stats

# Search from the command line
uv run python -m index.search "supercharged end-to-side AIN to ulnar transfer" -k 8

# Quality control and a per-article status export
uv run python -m harvester.report coverage
uv run python -m harvester.report sample --order random
uv run python -m harvester.report export          # -> data/status.csv

uv run pytest -q

Every step can be interrupted and restarted: status is kept per article in SQLite, and finished search windows are logged. Do not run two indexing processes at once; LanceDB allows one writer per table.

Connect it to an MCP client

The server speaks stdio. Add it to your client's MCP configuration, for example:

"nerve-lit": {
  "command": "<repo>/.venv/Scripts/python.exe",
  "args": ["<repo>/mcp-server/server.py"]
}

Keep the tool names as they are: they start with a read verb (search_, get_) so that a client's permission layer can treat them as read-only. tests/test_mcp_server.py guards that contract.

Performance notes

  • Warm search: ~0.2 s query embedding + ~0.1 s kNN + ~1.5 s reranking of 50 candidates on an RTX 3070.

  • First search after start-up: ~30 s (reranker weights and LanceDB pages come off disk). The MCP server warms this in a background thread at start.

  • On Windows, PyPI serves a CPU-only torch; pyproject.toml pins torch to the CUDA index. Check with uv run python -c "import torch; print(torch.cuda.is_available())".

Licence and data

Code: MIT. Article text is not part of this repository. Open-access articles carry their own licences (PMC OA subset, publisher OA terms); the pipeline fetches them only through NCBI and Unpaywall and keeps them local. Paywalled articles are never downloaded automatically.

Available Tools

3 tools
get_nerve_lit_articleB

Haal de volledige metadata van één artikel op via zijn PMID.

Inputs: pmid (required). Returns: result

get_nerve_lit_statsC

Toon de omvang en toestand van de literatuurdatabank.

Inputs: no named parameters. Returns: result

search_nerve_litA

Doorzoek de peripheral-nerve-surgery literatuurdatabank semantisch.

Args:
    query: de vraag in natuurlijke taal, bv. "supercharged end-to-side AIN naar ulnaris".
    k: aantal passages om terug te geven (default 8).
    year_from: enkel artikels vanaf dit publicatiejaar.
    year_to: enkel artikels tot en met dit publicatiejaar.
    full_text_only: alleen passages uit volledige artikels, niet uit abstracts.

Returns:
    De relevantste passages met hun citaat (titel, tijdschrift, jaar, PMID, DOI).

Inputs: query (required), k, year_to, year_from, +1 more. Returns: result

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.6/5.0

    Scored across 3 tools

    Disambiguation5/5

    The three tools map to clearly distinct operations: semantic passage search, single-article metadata retrieval by PMID, and database statistics. An agent can trivially choose between searching, fetching a known article, and inspecting corpus coverage, since the inputs (free-text query vs. PMID) also differ.

    Naming Consistency5/5

    All three names use the same snake_case verb_noun pattern with a shared 'nerve_lit' domain token (search_/get_/get_). The convention is predictable and unambiguous.

    Tool Count4/5

    Three tools is on the lean side but fits a narrowly scoped, read-only literature search service where each tool earns its place. No filler or redundant tools are present.

    Completeness4/5

    The surface covers the core lifecycle of a literature database: discover passages, retrieve full metadata, and check corpus status, with useful filters (year range, full-text only). Minor gaps remain, such as lookup or export by DOI and batch/citation retrieval, but these are workarounds rather than blockers.

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • F
      license
      Not graded
      quality
      D
      maintenance
      Provides seamless access to over 35 million PubMed scientific articles through natural language queries for research discovery and analysis. It enables tools for advanced searches, retrieving article details, and exploring related research within the life sciences and biomedical fields.
      13
      -
    • A
      license
      Not graded
      quality
      D
      maintenance
      Enables AI assistants to search, access, and analyze PubMed articles through MCP, providing tools for keyword and advanced search, article metadata retrieval, full-text PDF download, and deep paper analysis.
      MIT