AI Papers Helper

Search, index, and read academic AI/ML papers — from the terminal or directly inside your AI coding assistant.

AI Papers Helper crawls top-tier AI/ML conference proceedings, indexes them in a local SQLite database with FTS5 full-text search, queries arxiv, and converts paper PDFs into clean markdown for in-depth reading. It ships as both a CLI (papers) and an MCP server (ai-papers-mcp), so you can use it standalone or wire it into Claude Code / any MCP-compatible client.

✨ Features

• 🌐 Conference crawling — Fetch papers from NeurIPS, ICML, ICLR, CVPR, ICCV, WACV, AAAI, IJCAI, MLSys, ACL, EMNLP with parallel workers and incremental crawl state.

• 🔎 Local full-text search — SQLite + FTS5 with BM25 ranking (title-weighted) and relevance/date ordering. Missing-abstract penalty keeps results meaningful.

• 📡 arxiv search — Live arxiv search via web scraping (default) or the Atom API backend, with date filters (past 12 months, specific year, date range).

• 📄 PDF → Markdown — Parse papers through the MinerU API with content-addressable caching, so each PDF is only parsed once.

• 📑 TOC & grep — Extract a paper's table of contents and grep specific sections/equations from its full markdown.

• 🤝 MCP server — Expose everything as 4 tools to Claude Code or any MCP client: search the local library, search arxiv, get a paper's TOC, and grep paper content.

• 🧠 Smart resolution — Paper lookup falls back progressively: exact DB match → fuzzy match (SequenceMatcher, 0.8 threshold) → arxiv search.

Related MCP server: arXiv MCP Server

📦 Installation

Requires Python 3.12+.

# Clone
git clone https://github.com/<your-org>/ai_papers_helper.git
cd ai_papers_helper

# Install as a global tool (exposes `papers` and `ai-papers-mcp` on PATH)
uv tool install --force .

This exposes two console scripts:

Environment variables

All data lives under ~/.ai_papers_helper/:

~/.ai_papers_helper/
├── papers.db            # SQLite database (FTS5 index)
├── crawl_state.json     # Incremental crawl state
├── cache/               # HTTP response cache
├── arxiv_cache/         # arxiv result cache (24h TTL)
└── papers/              # Parsed markdown (content-addressed by URL hash)

🚀 Quick start

1. Initialize the database

papers init

2. Crawl conference papers

# Crawl everything new (incremental — skips already-crawled years)
papers update

# Crawl a specific conference and year
papers update --conference acl,emnlp --year 2024

# Force re-crawl a specific source/year
papers update --force --conference cvpr --year 2023

# Tune parallelism
papers update --workers 16

3. Search

# Search the local library (default: titles only)
papers search-library "diffusion model"
papers sl "graph neural network" --order-by date

# Titles + full abstracts
papers sl "transformer attention" --full-abs

# Paginate
papers sl "reinforcement learning" --page 2
papers sl "reinforcement learning" --from 31

# Search arxiv
papers search-arxiv "mixture of experts"
papers sa "vision transformer" --sort-by date --date-filter-by past_12
papers sa "llm" --date-filter-by specific_year --date-year 2024
papers sa "diffusion" --date-filter-by date_range --date-from 2024-01 --date-to 2024-06

4. Read a paper

# Get the full markdown (creates a /tmp/<title>.md symlink to the cached file)
papers content "Attention Is All You Need"

# Show the table of contents
papers content "Attention Is All You Need" --toc

🧩 Skill (for AI coding agents)

The repo ships a ready-made Agent Skills skill at papers-skill/SKILL.md. It teaches AI coding agents (pi, Claude Code, etc.) how to drive the papers CLI: when to prefer the local library vs arxiv, the search -> content -> grep workflow, and the full option reference.

The agent loads the skill on-demand when a task matches, then runs papers itself via the shell - no server process required. This is the lightest-weight way to let an agent search and read papers.

Install the skill

Point your agent at the papers-skill directory. For pi:

# Global (available in every project)
ln -s "$(pwd)/papers-skill" ~/.pi/agent/skills/papers

# Or project-level
mkdir -p .pi/skills && ln -s "$(pwd)/papers-skill" .pi/skills/papers

Skill vs MCP: The skill is just instructions (the agent runs the CLI via shell); the MCP server below exposes typed tools. The skill needs nothing running, the MCP server gives more structured tool calls - pick whichever fits your agent.

🤖 Using the MCP server

The same functionality is exposed as an MCP server for use inside Claude Code or any MCP-compatible client.

4 tools

🏗 Architecture

src/ai_papers_helper/
├── cli.py                 # Typer CLI: init, search-library, search-arxiv, update, content
├── mcp_server.py          # FastMCP server exposing 4 tools
├── config.py              # Paths, env vars, page-size constants
├── core/
│   ├── models.py          # Pydantic v2: Author, Paper
│   └── database.py        # SQLite + FTS5 singleton, BM25 ranking, auto-sync triggers
├── crawler/
│   ├── base.py            # BaseCrawler ABC + parallel detail-page fetching
│   ├── http.py            # Shared requests.Session w/ retry + file cache
│   ├── state.py           # CrawlState (per-source crawled years, JSON)
│   ├── cvf.py             # CVPR / ICCV / WACV
│   ├── aaai.py            # AAAI
│   ├── ijcai.py           # IJCAI
│   ├── icml.py            # ICML URL helpers
│   ├── acl_anthology.py   # ACL / EMNLP (ACL Anthology)
│   └── json_api.py        # Generic JSON API crawler (NeurIPS, ICML, ICLR, MLSys)
├── search/
│   ├── library_search.py  # Local FTS5 search, relevance/date ordering
│   └── arxiv_search.py    # arxiv dispatcher (web vs api backend)
├── retrieval/
│   ├── resolver.py        # Progressive lookup: exact → fuzzy → arxiv
│   ├── parser.py          # MinerU API client (async polling, content-addressed cache)
│   ├── content.py         # Markdown TOC extraction + section slicing
│   └── lookup.py          # End-to-end: title → paper → markdown
└── helper/                # arxiv web/api internals, pagination, rate limiting

How search works

• Local library: FTS5 with porter unicode61 tokenizer. BM25 with title weight 10.0, abstract weight 1.0. Results with missing abstracts are penalized (* 0.9) so well-documented papers surface first.

• Date ordering: BM25 rank is bucketed into relevance tiers; within a tier, newer papers come first — so you don't lose relevance entirely.

• arxiv: Web scraping by default (no API key, gentler). Switch to the Atom API with ARXIV_BACKEND=api for query-syntax power (field prefixes, boolean operators).

How PDF reading works

1. Resolve the paper by title (DB exact → fuzzy → arxiv).

2. Resolve a PDF URL (paper.pdf_url, else arxiv lookup, backfilling the DB).

3. Send to MinerU; poll until done; download & unzip the result.

4. Cache under ~/.ai_papers_helper/papers/{sha256(url)[:16]}/full.md — content-addressed, so re-reads are instant.

🧪 Development

uv sync                 # install deps
pytest                  # run the full suite
pytest tests/test_database.py          # single file
pytest -k "fuzzy"                      # by name pattern

Tests use temp databases, mock network calls (patch.object(crawler, "_fetch_url", ...)), and HTML/JSON fixtures in tests/fixtures/. See CLAUDE.md for the full contributor guide.

Conventions

• Python 3.12+, from __future__ import annotations in every file

• No async; parallelism via ThreadPoolExecutor

• Standard-library sqlite3 (no ORM), requests for HTTP

• Logging via logging.getLogger(__name__)

📄 License

MIT