Zotero Chunk RAG

Passage-level semantic search over a Zotero library. PDFs are extracted, split into overlapping text chunks, embedded via Gemini, and stored in ChromaDB. An MCP server exposes the index to Claude Code (or any MCP client) as tool calls that return specific passages with expandable surrounding context.

Install

Requires Python 3.10+ and a Gemini API key.

Setup

1. Configuration

Copy the example config and fill in your API key:

Edit ~/.config/zotero-chunk-rag/config.json:

Alternatively, set GEMINI_API_KEY as an environment variable and omit the field from the file.

Configuration reference

Embedding dimensions

gemini-embedding-001 supports output dimensions from 64 to 3072. The default 768 is a good general-purpose setting.

• Lower values (256-512): Faster search, smaller index on disk, slightly less precise retrieval. Suitable if your library is small or queries are topically broad.

• Higher values (1024-3072): More expressive embeddings that can distinguish finer semantic differences. Diminishing returns above 1024 for most academic search tasks, and the index grows linearly with dimension count.

Changing this value after indexing requires a full re-index (--force), since stored vectors must all share the same dimensionality.

Chunk size and overlap

chunk_size controls how much text (in approximate tokens) each passage contains. chunk_overlap controls how many tokens are shared between consecutive chunks.

• Smaller chunks (200-300 tokens): Each result is a tighter, more focused passage. Better for locating specific claims or definitions. Risk: a single idea may be split across chunks, reducing retrieval quality for broader questions.

• Larger chunks (500-800 tokens): Each result carries more context. Better for questions about methods or arguments that span multiple sentences. Risk: irrelevant surrounding text dilutes the embedding, making it harder to match precise queries.

• Overlap ensures that sentence-boundary content is not lost between chunks. Setting overlap to 25-30% of chunk size (the default ratio of 100/400) is a reasonable starting point. Too little overlap risks splitting key sentences at chunk boundaries; too much wastes embedding API calls on redundant text.

Like embedding dimensions, changing chunk size or overlap requires a full re-index.

2. Initial indexing

Index your full Zotero library (processes all PDFs, skips items without PDF attachments):

To test with a subset first:

This reads the Zotero SQLite database (read-only; safe to run while Zotero is open), extracts text from each PDF, chunks it, embeds via Gemini, and stores everything in ChromaDB.

3. Register the MCP server

Add to your Claude Code settings (~/.claude/settings.json) under mcpServers:

Restart Claude Code. The six tools (search_papers, search_topic, search_tables, get_passage_context, get_reranking_config, get_index_stats) will be available in every session.

User Guide

Updating from Zotero

Re-run the indexing script whenever you add new papers or attach PDFs to existing items:

The indexer is incremental by default. It compares the set of Zotero items that have PDFs against the set of document IDs already in the vector store, and only processes the difference. This means:

• New items with PDFs are indexed.

• Existing items that previously had no PDF but now do are indexed.

• Already-indexed items are skipped entirely (no re-embedding, no API cost).

To force a complete re-index (for example after changing chunk_size or embedding_dimensions):

Result reranking

Search results are reranked using a composite score that combines semantic similarity with document metadata:

Where:

• similarity: Cosine similarity from vector search (0-1)

• α (alpha): Exponent that compresses similarity range (default 0.7). Lower values give metadata more influence.

• section_weight: Weight based on document section (Results: 1.0, Methods: 0.85, Introduction: 0.5, etc.)

• journal_weight: Weight based on SCImago journal quartile (Q1: 1.0, Q2: 0.85, Q3: 0.65, Q4: 0.45)

This reranking ensures that:

1. Results from high-impact sections (Results, Conclusions) rank higher than boilerplate (Introduction, References)

2. Papers from higher-ranked journals receive a boost

3. Semantic relevance remains the primary factor, with metadata providing tiebreaking

Reranking can be disabled by setting "rerank_enabled": false in config.json.

Default section weights

Override any weight by passing section_weights={"section_name": weight} to search tools.

Understanding scores

• relevance_score (0-1): Raw cosine similarity. Higher = more semantically similar to query.

• composite_score (0-1): Weighted product of similarity, section weight, and journal weight.

A result with relevance_score=0.8, section=results (1.0), journal_quartile=Q1 (1.0) would have:

A result with relevance_score=0.8, section=introduction (0.5), journal_quartile=Q4 (0.45) would have:

Asking questions

The MCP server exposes six tools to Claude Code:

search_topic

Find the most relevant papers for a topic, deduplicated by document. Each paper is scored by both its average chunk relevance (overall topical fit) and its best single chunk (strongest individual passage). Results are sorted by average composite score.

Parameters:

• query (string, required) -- Natural language topic description.

• num_papers (int, default 10) -- Number of distinct papers to return (max 50).

• year_min / year_max (int, optional) -- Filter by publication year range.

• section_weights (dict, optional) -- Override default section weights.

• journal_weights (dict, optional) -- Override default journal quartile weights. Use "unknown" for papers without quartile data.

Each result includes:

• doc_title, authors, year, publication -- Bibliographic metadata.

• citation_key -- BetterBibTeX citation key for LaTeX \cite{}.

• journal_quartile -- SCImago quartile (Q1/Q2/Q3/Q4) or null.

• avg_score -- Average raw similarity across all matching chunks.

• best_chunk_score -- Raw similarity of the single most relevant chunk.

• avg_composite_score -- Average reranked score across matching chunks.

• best_composite_score -- Reranked score of the best chunk.

• best_passage_section -- Section label of the best matching passage.

• num_relevant_chunks -- How many chunks in this paper matched.

• best_passage, best_passage_page, best_passage_context -- The strongest passage with context.

search_papers

Passage-level semantic search across all indexed chunks. Returns the matching text, relevance score, document metadata, and optionally surrounding chunks for context.

Parameters:

• query (string, required) -- Natural language search query.

• top_k (int, default 10) -- Number of chunk results to return (max 50).

• context_chunks (int, default 1) -- How many adjacent chunks to include before and after each hit (0-3).

• year_min / year_max (int, optional) -- Filter results by publication year range.

• section_weights (dict, optional) -- Override default section weights. Set a section to 0 to exclude it.

• journal_weights (dict, optional) -- Override default journal quartile weights.

Each result includes:

• passage -- The matched chunk text.

• context_before / context_after -- Lists of adjacent chunk texts.

• full_context -- All context merged into a single string.

• doc_title, authors, year, publication, page -- Bibliographic metadata.

• citation_key -- BetterBibTeX citation key.

• doc_id, chunk_index -- Identifiers for follow-up with get_passage_context.

• relevance_score -- Raw cosine similarity (0-1).

• composite_score -- Reranked score incorporating section and journal weights.

• section -- Document section label (abstract, methods, results, etc.).

• section_confidence -- Confidence of section detection (0-1).

• journal_quartile -- SCImago quartile (Q1/Q2/Q3/Q4) or null.

get_passage_context

Expands the context window around a specific passage returned by search_papers. Use this when the initial result is relevant but you need to read more of the surrounding text.

Parameters:

• doc_id (string, required) -- Document ID from a search result.

• chunk_index (int, required) -- Chunk index from a search result.

• window (int, default 2) -- Chunks before/after to include (1-5).

Returns:

• doc_id, doc_title, citation_key -- Document identifiers.

• section -- Section label of the center chunk.

• section_confidence -- Confidence of section detection.

• journal_quartile -- SCImago quartile or null.

• center_chunk_index, window -- Position and expansion parameters.

• passages -- List of chunks, each with chunk_index, page, section, text, is_center.

• merged_text -- All passages concatenated.

get_reranking_config

Returns the current reranking configuration: section weights, journal quartile weights, alpha exponent, and valid section/quartile names. Useful for understanding how results are ranked and what override values are valid.

get_index_stats

Returns the current state of the index: total documents, total chunks, and average chunks per document. No parameters. Useful for verifying that indexing completed and checking coverage.

Example session

In Claude Code, ask questions that reference your library. Claude will call the tools automatically when the MCP server is registered:

OCR for scanned PDFs

For PDFs that contain scanned images instead of text (common with older papers), enable OCR to extract searchable text:

1. Install Tesseract OCR engine:

   • Windows: Download from https://github.com/UB-Mannheim/tesseract/wiki

   • macOS: brew install tesseract

   • Linux: sudo apt install tesseract-ocr

2. Install Python bindings:

   pip install pytesseract Pillow

3. Enable in config.json:

   { "ocr_enabled": true, "ocr_language": "eng", "ocr_dpi": 300 }

   Or use the CLI flag:

   python scripts/index_library.py --ocr

OCR settings:

Notes:

• OCR is only applied to pages that lack extractable text

• Processing time increases significantly with OCR enabled

• Re-index with --force after enabling OCR to process previously skipped PDFs

Table extraction

Extract and search tables from PDFs separately from text:

Or use CLI:

Table settings:

Searching tables:

Use the search_tables tool to find tables by content:

Parameters:

• query (string, required) -- Natural language search query for table content.

• top_k (int, default 10) -- Number of tables to return (max 30).

• year_min / year_max (int, optional) -- Filter by publication year range.

• journal_weights (dict, optional) -- Override default journal quartile weights.

Each result includes:

• table_markdown -- Full table as markdown

• caption -- Table caption if detected

• num_rows, num_cols -- Table dimensions

• doc_title, authors, year, citation_key -- Bibliographic info

• journal_quartile -- SCImago quartile (Q1/Q2/Q3/Q4) or null

• page -- Page number where table appears

• relevance_score -- Semantic similarity to query (0-1)

• composite_score -- Reranked score incorporating section (0.9 for tables) and journal weights

Notes:

• Requires PyMuPDF 1.23+ for find_tables() support

• Tables are stored as markdown for semantic search

• Captions are detected if they follow "Table N:" pattern

• Use get_passage_context with table_page and table_index to find text referencing a table

Integration with agents and skills

The zotero-research skill is an example Claude Code agent designed to use this server. It accepts high-level research requests ("find 10 papers about X", "find citations for and against Y") and returns formatted markdown with verbatim passages, summaries, and BetterBibTeX citation keys. See examples/zotero-research/SKILL.md for the full skill specification.